Tweaking PEP 8 guidelines for use of leading underscores
Currently, the naming section of PEP 8 doesn't say very much about what a leading underscore *means* in the Python standard library. I would like to add a new "Private interfaces" subsection under "Naming Conventions" to say the following: ================= Private interfaces Unless explicitly documented otherwise, a leading underscore on any name indicates that it is an internal implementation detail and backwards compatibility guarantees do not apply. It is strongly encouraged that private APIs (whether modules, classes, functions, attributes or other names) be clearly marked in this way, as many Python users rely on introspection to identify available functionality and may be mislead into believing an API without the leading underscore is in fact a public API with the standard backwards compatibility guarantees. All test modules are also considered private interfaces. Even though they typically lack the leading underscore, modules imported by another module are also considered an implementation detail. Other modules *should* not rely on indirect access to such modules unless they are an explicitly documented part of the API (such as ``os.path``). ================= Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 14 July 2013 18:11, Nick Coghlan <ncoghlan@gmail.com> wrote:
Currently, the naming section of PEP 8 doesn't say very much about what a leading underscore *means* in the Python standard library.
I would like to add a new "Private interfaces" subsection under "Naming Conventions" to say the following:
================= Private interfaces
Unless explicitly documented otherwise, a leading underscore on any name indicates that it is an internal implementation detail and backwards compatibility guarantees do not apply. It is strongly encouraged that private APIs (whether modules, classes, functions, attributes or other names) be clearly marked in this way, as many Python users rely on introspection to identify available functionality and may be mislead into believing an API without the leading underscore is in fact a public API with the standard backwards compatibility guarantees.
All test modules are also considered private interfaces.
Even though they typically lack the leading underscore, modules imported by another module are also considered an implementation detail. Other modules *should* not rely on indirect access to such modules unless they are an explicitly documented part of the API (such as ``os.path``). =================
Slight adjustment to the proposed wording to ensure completely undocumented modules are also considered private: ================= Private interfaces Unless explicitly documented otherwise, a leading underscore on any name indicates that it is an internal implementation detail and backwards compatibility guarantees do not apply. It is strongly encouraged that private APIs (whether modules, classes, functions, attributes or other names) be clearly marked in this way, as Python users may rely on introspection to identify available functionality and may be misled into believing an API without a leading underscore is in fact a public API with the standard backwards compatibility guarantees. Even when their names do not start with a leading underscore, all test modules and all modules that are not covered in the documentation are also considered private interfaces. Similarly, the specific modules and external APIs imported by a module are always considered an implementation detail. Other modules should not rely on indirect access to such imported interfaces unless they are an explicitly documented part of the containing module's API (such as ``os.path`` or a package ``__init__`` module exposing functionality from submodules). ================= -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Jul 14, 2013 at 7:09 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 14 July 2013 18:11, Nick Coghlan <ncoghlan@gmail.com> wrote:
Currently, the naming section of PEP 8 doesn't say very much about what a leading underscore *means* in the Python standard library.
I would like to add a new "Private interfaces" subsection under "Naming Conventions" to say the following:
================= Private interfaces
Unless explicitly documented otherwise, a leading underscore on any name indicates that it is an internal implementation detail and backwards compatibility guarantees do not apply. It is strongly encouraged that private APIs (whether modules, classes, functions, attributes or other names) be clearly marked in this way, as many Python users rely on introspection to identify available functionality and may be mislead into believing an API without the leading underscore is in fact a public API with the standard backwards compatibility guarantees.
All test modules are also considered private interfaces.
Even though they typically lack the leading underscore, modules imported by another module are also considered an implementation detail. Other modules *should* not rely on indirect access to such modules unless they are an explicitly documented part of the API (such as ``os.path``). =================
Slight adjustment to the proposed wording to ensure completely undocumented modules are also considered private:
================= Private interfaces
Unless explicitly documented otherwise, a leading underscore on any name indicates that it is an internal implementation detail and backwards compatibility guarantees do not apply. It is strongly encouraged that private APIs (whether modules, classes, functions, attributes or other names) be clearly marked in this way, as Python users may rely on introspection to identify available functionality and may be misled into believing an API without a leading underscore is in fact a public API with the standard backwards compatibility guarantees.
Even when their names do not start with a leading underscore, all test modules and all modules that are not covered in the documentation are also considered private interfaces.
Similarly, the specific modules and external APIs imported by a module are always considered an implementation detail. Other modules should not rely on indirect access to such imported interfaces unless they are an explicitly documented part of the containing module's API (such as ``os.path`` or a package ``__init__`` module exposing functionality from submodules). =================
+1
+1 This is already how we've been behaving for years. On Sun, Jul 14, 2013 at 4:09 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 14 July 2013 18:11, Nick Coghlan <ncoghlan@gmail.com> wrote:
Currently, the naming section of PEP 8 doesn't say very much about what a leading underscore *means* in the Python standard library.
I would like to add a new "Private interfaces" subsection under "Naming Conventions" to say the following:
Slight adjustment to the proposed wording to ensure completely undocumented modules are also considered private:
================= Private interfaces
Unless explicitly documented otherwise, a leading underscore on any name indicates that it is an internal implementation detail and backwards compatibility guarantees do not apply. It is strongly encouraged that private APIs (whether modules, classes, functions, attributes or other names) be clearly marked in this way,
<snip>
as Python users may rely on introspection to identify available functionality and may be misled into believing an API without a leading underscore is in fact a public API with the standard backwards compatibility guarantees.
While true, I'm not sure the last part of the sentence is necessary. Once we've established that a leading _ indicates something is private there isn't much point in explaining the various ways people might find them. I'm happy regardless of this bit being there.
Even when their names do not start with a leading underscore, all test modules and all modules that are not covered in the documentation are also considered private interfaces.
Similarly, the specific modules and external APIs imported by a module are always considered an implementation detail. Other modules should not rely on indirect access to such imported interfaces unless they are an explicitly documented part of the containing module's API (such as ``os.path`` or a package ``__init__`` module exposing functionality from submodules). =================
On 15 July 2013 02:15, Gregory P. Smith <greg@krypto.org> wrote:
On Sun, Jul 14, 2013 at 4:09 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
as Python users may rely on introspection to identify available functionality and may be misled into believing an API without a leading underscore is in fact a public API with the standard backwards compatibility guarantees.
While true, I'm not sure the last part of the sentence is necessary. Once we've established that a leading _ indicates something is private there isn't much point in explaining the various ways people might find them. I'm happy regardless of this bit being there.
You'd be surprised how many non-core devs react with astonishment when I suggest that not documenting something isn't enough to avoid having users consider it a supported public API - they usually get it after I point out how far you can usually get just by using dir() and help() to play with a new library at the interactive prompt instead of looking at out-of-band docs. I figure including the second part will help prevent some "But why?" reactions in the future. I think Steven has a reasonable point about being clearer that an explicit leading underscore is the preferred solution for any new private modules, though, so here's an updated proposal: ================= Private interfaces Unless explicitly documented otherwise, a leading underscore on any name indicates that it is an internal implementation detail and any backwards compatibility guarantees do not apply. It is strongly encouraged that private APIs (whether modules, classes, functions, attributes or other names) be clearly marked in this way, as Python users may rely on introspection to identify available functionality and may be misled into believing an API without a leading underscore is in fact a public API with the standard backwards compatibility guarantees. While the explicit use of a leading underscore in the names of private modules is preferred, all test modules and all modules that are not explicitly covered in the documentation are also considered private interfaces, even when their names do not start with a leading underscore and even if they include a module level documentation string. This includes submodules of packages that are documented as if they were a single module. Similarly, the specific modules and external APIs imported by a module are always considered an implementation detail. Other modules should not rely on indirect access to such imported interfaces unless they are an explicitly documented part of the containing module's API (such as ``os.path`` or a package ``__init__`` module that exposes functionality from submodules). ================= -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Jul 15, 2013 at 6:17 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
================= Private interfaces
Unless explicitly documented otherwise, a leading underscore on any name indicates that it is an internal implementation detail and any backwards compatibility guarantees do not apply. It is strongly encouraged that private APIs (whether modules, classes, functions, attributes or other names) be clearly marked in this way, as Python users may rely on introspection to identify available functionality and may be misled into believing an API without a leading underscore is in fact a public API with the standard backwards compatibility guarantees.
Very good.
While the explicit use of a leading underscore in the names of private modules is preferred, all test modules and all modules that are not explicitly covered in the documentation are also considered private interfaces, even when their names do not start with a leading underscore and even if they include a module level documentation string. This includes submodules of packages that are documented as if they were a single module.
But wait, aren't this about how to use other peoples modules, more than a style guide of how to write your own modules? I don't think this belongs in PEP 8 at all. //Lennart
On 7/15/2013 12:17 AM, Nick Coghlan wrote:
You'd be surprised how many non-core devs react with astonishment when I suggest that not documenting something isn't enough to avoid having users consider it a supported public API - they usually get it after I point out how far you can usually get just by using dir() and help() to play with a new library at the interactive prompt instead of looking at out-of-band docs. I figure including the second part will help prevent some "But why?" reactions in the future.
I agree with the goal of preventing such reactions, but I suggest slightly different means to do so. I just looked and idlelib.__init__.py consists of """# Dummy file to make this a package. """ help(idlelib) prints the comment (which surprised me, but it seems to be smart enough to look for a 'doc comment' when it does not find a doc string). I now plan to change the comment to a docstring so 'idlelib.__doc__' also works and change the text to something like """A package of private modules that implement the IDLE integrated shell and editor application. See the manual for the public entry points.""" (or maybe list them briefly here in the doc string). This should properly inform people who introspect. Also, the manual needs a small, indexed, section on idlelib that says much the same thing along with documenting what *is* public (which is not completely clear to me at the moment).
I think Steven has a reasonable point about being clearer that an explicit leading underscore is the preferred solution for any new private modules, though, so here's an updated proposal:
The public/private distinction is not always as clean-cut as seems to be assumed in this discussion. Test and idlelib both have public entry points to an application based on private files. ('Running the Python test suite' is a batch application.) So I do not think either should be _ prefixed. For both, and anything similar, a proper docstring and manual entry explaining the nuances should be completely acceptable as a substitute for a blunt, all-or-nothing flag. I also think that marking something private, either way, or partially private with words, should extend to its contents. In particular, the 100s of implementation files in test and idlelib need not be _ marked as as they would be covered once test and idlelib are properly documented. The special _ mark makes sense when it is the exception, but not in a context where it would be the overwhelming norm, because everything but one or two files is private.
================= Private interfaces
Unless explicitly documented otherwise, a leading underscore on any name indicates that it is an internal implementation detail and any backwards compatibility guarantees do not apply. It is strongly encouraged that private APIs (whether modules, classes, functions, attributes or other names) be clearly marked in this way, as Python users may rely on introspection to identify available functionality and may be misled into believing an API without a leading underscore is in fact a public API with the standard backwards compatibility guarantees.
While the explicit use of a leading underscore in the names of private modules is preferred, all test modules and all modules that are not explicitly covered in the documentation are also considered private interfaces, even when their names do not start with a leading underscore and even if they include a module level documentation string. This includes submodules of packages that are documented as if they were a single module.
Since I am planning to explicitly cover idlelib in the doc, to list the public interface, just as is done for test, I would like it explicitly mentioned along with test. I do not think absence of documentation is a good signal. I would rather say that any private or partially private package that does not have a _ name *must* have a docstring saying that it is all or mostly private. (Or be a subpackage or module of something so marked.) And presuming that it is associated with something that is documented, then the doc should state that the implementation files are private. In other words, I suggest that "modules that are not explicitly covered in the documentation ... [whose] names do not start with a leading underscore" should be empty. And I think "even if they include a module level documentation string." is backwards. Such files *should* have a docstring that serves as a substitute for a _ prefix.
Similarly, the specific modules and external APIs imported by a module are always considered an implementation detail. Other modules should not rely on indirect access to such imported interfaces unless they are an explicitly documented part of the containing module's API (such as ``os.path`` or a package ``__init__`` module that exposes functionality from submodules). =================
-- Terry Jan Reedy
On 15 July 2013 17:31, Terry Reedy <tjreedy@udel.edu> wrote:
I do not think absence of documentation is a good signal. I would rather say that any private or partially private package that does not have a _ name *must* have a docstring saying that it is all or mostly private. (Or be a subpackage or module of something so marked.) And presuming that it is associated with something that is documented, then the doc should state that the implementation files are private.
In other words, I suggest that "modules that are not explicitly covered in the documentation ... [whose] names do not start with a leading underscore" should be empty. And I think "even if they include a module level documentation string." is backwards. Such files *should* have a docstring that serves as a substitute for a _ prefix.
I (now) agree - a disclaimer in the docstring is a better signal than an absence of documentation, and one we can reasonably make a requirement even for external applications that we bundle or otherwise bless as the "obvious way to do it" (as is happening with pip). ================= Private interfaces Unless explicitly documented otherwise, a leading underscore on any name indicates that it is an internal implementation detail and any backwards compatibility guarantees do not apply. It is strongly encouraged that private APIs (whether modules, classes, functions, attributes or other names) be clearly marked in this way, as Python users may rely on introspection to identify available functionality and may be misled into believing an API without a leading underscore is in fact a public API with the standard backwards compatibility guarantees. While the explicit use of a leading underscore is the preferred solution, the names of some private (or partially private) modules (such as ``test`` and ``idlelib``) lack the leading underscore either for historical reasons or because they expose a public command line interface through the ``-m`` switch. Such modules should include an explicit disclaimer in their module docstring, indicating that they do not use the leading underscore convention and noting where the definition of the public API (if any) can be found (the public API definition may also be part of the module docstring). As a general principle, the specific modules and external APIs imported by a module are always considered an implementation detail. Other modules should not rely on indirect access to such imported interfaces unless they are an explicitly documented part of the containing module's API (such as ``os.path`` or a package ``__init__`` module that exposes functionality from submodules). ================= Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Working from what I think is the latest version. In general, i'd rather be prescriptive of future conventions than descriptive of current conventions. It's okay to exempt existing code, and as a general rule we've never been fond of rewriting existing code to update it to new standards or APIs. We don't need to do so here either. On Jul 15, 2013, at 05:48 PM, Nick Coghlan wrote:
Private interfaces
"Internal" or "Non-public"
Unless explicitly documented otherwise, a leading underscore on any name indicates that it is an internal implementation detail and any backwards compatibility guarantees do not apply. It is strongly encouraged that private APIs (whether modules, classes, functions, attributes or other names) be clearly marked in this way, as Python users may rely on introspection to identify available functionality and may be misled into believing an API without a leading underscore is in fact a public API with the standard backwards compatibility guarantees.
How about: "All internal interfaces (modules, classes, functions, attributes or other names) should be prefixed with a single leading underscore. Such names are internal implementation details for which no backward compatibility guarantees are made, unless otherwise specified. Existing code and other narrowly accepted exceptions may override this recommendation, in which case the docstrings and/or documentation for such code must clearly and explicitly state the internal status of the APIs. Imported names should always be considered an implementation detail. Other modules must not rely on indirect access to such imported names unless they are an explicitly documented part of the containing module's API, such as ``os.path`` or a package's ``__init__`` module that exposes functionality from submodules. Public names exported by a module should be include in the module's ``__all__`` attribute."
While the explicit use of a leading underscore is the preferred solution, the names of some private (or partially private) modules (such as ``test`` and ``idlelib``) lack the leading underscore either for historical reasons or because they expose a public command line interface through the ``-m`` switch. Such modules should include an explicit disclaimer in their module docstring, indicating that they do not use the leading underscore convention and noting where the definition of the public API (if any) can be found (the public API definition may also be part of the module docstring).
As a general principle, the specific modules and external APIs imported by a module are always considered an implementation detail. Other modules should not rely on indirect access to such imported interfaces unless they are an explicitly documented part of the containing module's API (such as ``os.path`` or a package ``__init__`` module that exposes functionality from submodules).
Cheers, -Barry
On Mon, 2013-07-15 at 18:40 -0400, Barry Warsaw wrote:
Working from what I think is the latest version.
In general, i'd rather be prescriptive of future conventions than descriptive of current conventions. It's okay to exempt existing code, and as a general rule we've never been fond of rewriting existing code to update it to new standards or APIs. We don't need to do so here either.
FWIW, I'm very skeptical of a PEP 8 guideline that would try to proscribe that the "non-internal" API of a module or class would be defined solely by a naming convention. If what's being described here does become a rule, there is reason to believe that future users who treat this PEP as the word-of-god (and there are a *lot* of them; I hear from people literally every week who want to "PEP8-ify" my code in some limited-value-added way) will be harmed. They'll be living in a fantasy world where every non-underscore-prefixed thing is now a defacto API. But I have lived in a world where that has not been the case since 1998, and the chance that I'll go back and change all my public code to satisfy a questionable introspection convention is pretty slim. - C
On Jul 15, 2013, at 05:48 PM, Nick Coghlan wrote:
Private interfaces
"Internal" or "Non-public"
Unless explicitly documented otherwise, a leading underscore on any name indicates that it is an internal implementation detail and any backwards compatibility guarantees do not apply. It is strongly encouraged that private APIs (whether modules, classes, functions, attributes or other names) be clearly marked in this way, as Python users may rely on introspection to identify available functionality and may be misled into believing an API without a leading underscore is in fact a public API with the standard backwards compatibility guarantees.
How about:
"All internal interfaces (modules, classes, functions, attributes or other names) should be prefixed with a single leading underscore. Such names are internal implementation details for which no backward compatibility guarantees are made, unless otherwise specified.
Existing code and other narrowly accepted exceptions may override this recommendation, in which case the docstrings and/or documentation for such code must clearly and explicitly state the internal status of the APIs.
Imported names should always be considered an implementation detail. Other modules must not rely on indirect access to such imported names unless they are an explicitly documented part of the containing module's API, such as ``os.path`` or a package's ``__init__`` module that exposes functionality from submodules. Public names exported by a module should be include in the module's ``__all__`` attribute."
While the explicit use of a leading underscore is the preferred solution, the names of some private (or partially private) modules (such as ``test`` and ``idlelib``) lack the leading underscore either for historical reasons or because they expose a public command line interface through the ``-m`` switch. Such modules should include an explicit disclaimer in their module docstring, indicating that they do not use the leading underscore convention and noting where the definition of the public API (if any) can be found (the public API definition may also be part of the module docstring).
As a general principle, the specific modules and external APIs imported by a module are always considered an implementation detail. Other modules should not rely on indirect access to such imported interfaces unless they are an explicitly documented part of the containing module's API (such as ``os.path`` or a package ``__init__`` module that exposes functionality from submodules).
Cheers, -Barry _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/lists%40plope.com
On 16/07/13 10:23, Chris McDonough wrote:
If what's being described here does become a rule, there is reason to believe that future users who treat this PEP as the word-of-god (and there are a *lot* of them; I hear from people literally every week who want to "PEP8-ify" my code in some limited-value-added way) will be harmed.
I sympathise with your pain, but is that not true of every PEP 8 naming convention?
They'll be living in a fantasy world where every non-underscore-prefixed thing is now a defacto API.
If your code has no obvious, documented convention at all for what's internal and what is not, they are no worse off. If you do have a documented convention for internal implementation details, then you are no worse off. "I have better things to do than PEP8-ify old, working, stable code" is a perfectly acceptable answer. "I have better things to do than PEP9-ify old, working, stable code, but if you want to provide regression tests and a working patch, I'll let you do so" might be an even better one :-) -- Steven
On Tue, 2013-07-16 at 11:25 +1000, Steven D'Aprano wrote:
If your code has no obvious, documented convention at all for what's internal and what is not, they are no worse off.
If you do have a documented convention for internal implementation details, then you are no worse off. "I have better things to do than PEP8-ify old, working, stable code" is a perfectly acceptable answer. "I have better things to do than PEP9-ify old, working, stable code, but if you want to provide regression tests and a working patch, I'll let you do so" might be an even better one :-)
Welp, I guess I'm logically boxed in then. Thanks for showing me the errors in my thinking. Should be no problem to manage the updating of that 500K lines of public code. /scarcasm - C
On 16 July 2013 12:20, Chris McDonough <chrism@plope.com> wrote:
On Tue, 2013-07-16 at 11:25 +1000, Steven D'Aprano wrote:
If your code has no obvious, documented convention at all for what's internal and what is not, they are no worse off.
If you do have a documented convention for internal implementation details, then you are no worse off. "I have better things to do than PEP8-ify old, working, stable code" is a perfectly acceptable answer. "I have better things to do than PEP9-ify old, working, stable code, but if you want to provide regression tests and a working patch, I'll let you do so" might be an even better one :-)
Welp, I guess I'm logically boxed in then. Thanks for showing me the errors in my thinking. Should be no problem to manage the updating of that 500K lines of public code.
/scarcasm
How do get from "If this doesn't apply to a module, just add something like 'This is an internal API' or 'This module includes internal APIs, consult the documentation for the public API' to the module docstring" to "updating 500k lines of public code"? The version in Barry's email that you replied to has that escape clause in it, so the fact it was missing from my original text doesn't justify this reaction. Cheers, Nick. P.S. Note that, while I'm trying to account for it in this particular case, we're never going to let the fact that many people misuse PEP 8 by considering it as a holy standard that should be followed by all Python code everywhere stop us from including updates that are valid specifically for the standard library. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, 2013-07-16 at 12:34 +1000, Nick Coghlan wrote:
How do get from "If this doesn't apply to a module, just add something like 'This is an internal API' or 'This module includes internal APIs, consult the documentation for the public API' to the module docstring" to "updating 500k lines of public code"? The version in Barry's email that you replied to has that escape clause in it, so the fact it was missing from my original text doesn't justify this reaction.
I may be wrong, but as I see it this is a decision to help (although only conditionally!) a theoretical IDE user or an agressive user of dir() at the expense of effectively either 1) making existing maintainers revisit existing code, or 2 ) making maintainers continually explain to users that their codebase is not "PEP8 compliant".
P.S. Note that, while I'm trying to account for it in this particular case, we're never going to let the fact that many people misuse PEP 8 by considering it as a holy standard that should be followed by all Python code everywhere stop us from including updates that are valid specifically for the standard library.
Fair enough, technically. But historical reality is that a lot of people just haven't read much past the title: "Style Guide for Python Code". Or at least they appear to have no concept that it's about code *in the stdlib only*. People take it way, way too seriously for non-stdlib code. But they do, and it takes time to manage that. Given that misunderstanding, is there a way to divorce stdlib-centric guidelines like the one being discussed now from "PEP8"-ness? I don't think any amount of marketing effort or reasoned explanation is going to separate "PEP8" from "correct code" in people's minds at this point. - C ps.. the real irritant behind my minirant is this: OSS developers have spent many months jumping through bw incompat hoops in Python over the last few years, and it has taken time away from doing things that provide value. The less I can do of that, the better, and Python gets more value too. That said, I realize that I'm in the minority because I happen to have a metric ton of public code out there. But it'd be nice if that was encouraged rather than effectively punished on the hunch that it might provide some benefit for a theoretical new user.
On 16 July 2013 13:02, Chris McDonough <chrism@plope.com> wrote:
OSS developers have spent many months jumping through bw incompat hoops in Python over the last few years, and it has taken time away from doing things that provide value. The less I can do of that, the better, and Python gets more value too. That said, I realize that I'm in the minority because I happen to have a metric ton of public code out there. But it'd be nice if that was encouraged rather than effectively punished on the hunch that it might provide some benefit for a theoretical new user.
You, Armin and everyone else that works on the bytes/text boundary are indeed the hardest hit by the Python 3 transition, and I appreciate the hard work you have all done to help make that transition as successful as it has been so far. However, the fact that people abuse PEP 8 by treating it as "all Python code in the world should follow these rules" cannot, and will not, stop us from continuing to use it to set appropriate guidelines *for the standard library*. I'll look into adding some stronger wording at the top making it clear that while PEP 8 is a useful starting point and a good default if a project doesn't have a defined style guide of it's own, it is *not* the be-all-and-end-all for Python style guides. Treating it as such as an abuse of the PEP, pure and simple. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, 2013-07-16 at 13:11 +1000, Nick Coghlan wrote:
On 16 July 2013 13:02, Chris McDonough <chrism@plope.com> wrote:
OSS developers have spent many months jumping through bw incompat hoops in Python over the last few years, and it has taken time away from doing things that provide value. The less I can do of that, the better, and Python gets more value too. That said, I realize that I'm in the minority because I happen to have a metric ton of public code out there. But it'd be nice if that was encouraged rather than effectively punished on the hunch that it might provide some benefit for a theoretical new user.
You, Armin and everyone else that works on the bytes/text boundary are indeed the hardest hit by the Python 3 transition, and I appreciate the hard work you have all done to help make that transition as successful as it has been so far.
However, the fact that people abuse PEP 8 by treating it as "all Python code in the world should follow these rules" cannot, and will not, stop us from continuing to use it to set appropriate guidelines *for the standard library*.
I'll look into adding some stronger wording at the top making it clear that while PEP 8 is a useful starting point and a good default if a project doesn't have a defined style guide of it's own, it is *not* the be-all-and-end-all for Python style guides. Treating it as such as an abuse of the PEP, pure and simple.
I understand that. Unfortunately the remainder of the world does not. The same IDEs that would be helped via this proposed change have "PEP8 modes" turned on *by default*! http://blog.jetbrains.com/pycharm/2013/02/long-awaited-pep-8-checks-on-the-f... It seems like an unwise step to continue stuffing things into the PEP8 brand bag as a result if that stuff is only meant to apply to the stdlib. - C
On 16 July 2013 13:16, Chris McDonough <chrism@plope.com> wrote:
I understand that. Unfortunately the remainder of the world does not. The same IDEs that would be helped via this proposed change have "PEP8 modes" turned on *by default*! http://blog.jetbrains.com/pycharm/2013/02/long-awaited-pep-8-checks-on-the-f...
It seems like an unwise step to continue stuffing things into the PEP8 brand bag as a result if that stuff is only meant to apply to the stdlib.
Clarifying what constitutes an internal interface in a way that doesn't require renaming anything is a necessary prerequisite for bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes its internal implemetnation API as "import pip" rather than "import _pip" and renaming it would lead to a lot of pointless code churn). Without that concern, the topic never would have come up. Yes, this new section may give the PEP 8 prescriptivists more ammunition to complain about existing projects with different coding standards not complying with the latest standard library guidelines. We're not unfamiliar with the complaint, given the number of standard library modules (some dating from the 20th century) that don't comply with the current incarnation of PEP 8 circa 2013. However, PEP 8 is already quite explicit about what we think of that kind of behaviour: http://www.python.org/dev/peps/pep-0008/#a-foolish-consistency-is-the-hobgob... PEP 8 is a tool. Like many of the other tools we provide as part of Python, it can be abused. However, the likelihood of such abuse is *not* an excuse to avoid using the tool appropriately, it's a reason to continue the thankless task of trying to educate people on what's appropriate and what isn't (and making wholesale changes to working code just to comply with a style guide generally isn't appropriate). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, 2013-07-16 at 15:44 +1000, Nick Coghlan wrote:
On 16 July 2013 13:16, Chris McDonough <chrism@plope.com> wrote:
I understand that. Unfortunately the remainder of the world does not. The same IDEs that would be helped via this proposed change have "PEP8 modes" turned on *by default*! http://blog.jetbrains.com/pycharm/2013/02/long-awaited-pep-8-checks-on-the-f...
It seems like an unwise step to continue stuffing things into the PEP8 brand bag as a result if that stuff is only meant to apply to the stdlib.
Clarifying what constitutes an internal interface in a way that doesn't require renaming anything is a necessary prerequisite for bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes its internal implemetnation API as "import pip" rather than "import _pip" and renaming it would lead to a lot of pointless code churn). Without that concern, the topic never would have come up.
Yes, this new section may give the PEP 8 prescriptivists more ammunition to complain about existing projects with different coding standards not complying with the latest standard library guidelines. We're not unfamiliar with the complaint, given the number of standard library modules (some dating from the 20th century) that don't comply with the current incarnation of PEP 8 circa 2013.
However, PEP 8 is already quite explicit about what we think of that kind of behaviour: http://www.python.org/dev/peps/pep-0008/#a-foolish-consistency-is-the-hobgob...
PEP 8 is a tool. Like many of the other tools we provide as part of Python, it can be abused. However, the likelihood of such abuse is *not* an excuse to avoid using the tool appropriately, it's a reason to continue the thankless task of trying to educate people on what's appropriate and what isn't (and making wholesale changes to working code just to comply with a style guide generally isn't appropriate).
Well, I know it's arguing with the umpire at this point, but PEP8 prescriptionism (great word by the way!) is de rigeur these days. It's a python-dev perogative to ignore that, but it has consequences that reverberate further than this maillist. - C
On 16 July 2013 16:01, Chris McDonough <chrism@plope.com> wrote:
Well, I know it's arguing with the umpire at this point, but PEP8 prescriptionism (great word by the way!) is de rigeur these days. It's a python-dev perogative to ignore that, but it has consequences that reverberate further than this maillist.
Yeah, we should probably be a bit more vocal in pointing out that PEP 8 is something to be applied judiciously, *not* treated as "sacred writ, never to be questioned". It's very tempting to just sigh and ignore it, instead of explicitly pointing out the problems with taking it too literally for people's personal projects (or, worse, using it to abuse developers of projects that long predate the recommendations in more recent versions) :P Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Jul 16, 2013, at 04:10 PM, Nick Coghlan wrote:
Yeah, we should probably be a bit more vocal in pointing out that PEP 8 is something to be applied judiciously, *not* treated as "sacred writ, never to be questioned". It's very tempting to just sigh and ignore it, instead of explicitly pointing out the problems with taking it too literally for people's personal projects (or, worse, using it to abuse developers of projects that long predate the recommendations in more recent versions) :P
In my experience, most PEP 8 violation bugs that I see for various upstreams (mine and others) comes from running the pep8 tool over the code. A lot of upstreams do this as part of their normal QA (e.g. you can set up unittests that run pep8, pyflakes, and other linters). Those will generally be fine because they're already committed to conformance to pep8's notion of what PEP 8 says <wink>[1]. In other cases, Someone Out There runs pep8 over your code, and files bugs on all the complaints that get produced. That can definitely lead to churn and waste, especially on big, established code bases. -Barry [1] which I do not always agree with anyway, as evidenced by the recent discussion on closing paren indentation for multi-line collections.
On 16/07/2013 6:44am, Nick Coghlan wrote:
Clarifying what constitutes an internal interface in a way that doesn't require renaming anything is a necessary prerequisite for bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes its internal implemetnation API as "import pip" rather than "import _pip" and renaming it would lead to a lot of pointless code churn). Without that concern, the topic never would have come up.
BTW, how does the use of __all__ effect things? Somewhere I got the idea that if a module uses __all__ then anything not listed is internal. I take it that is wrong? -- Richard
On 16 July 2013 20:28, Richard Oudkerk <shibturn@gmail.com> wrote:
On 16/07/2013 6:44am, Nick Coghlan wrote:
Clarifying what constitutes an internal interface in a way that doesn't require renaming anything is a necessary prerequisite for bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes its internal implemetnation API as "import pip" rather than "import _pip" and renaming it would lead to a lot of pointless code churn). Without that concern, the topic never would have come up.
BTW, how does the use of __all__ effect things? Somewhere I got the idea that if a module uses __all__ then anything not listed is internal. I take it that is wrong?
Gah, you're right. I did consider that, then forgot to include it in the proposed text. Rather than throwing more versions of the text at the list, I'll create a tracker issue and post a proposed patch for PEP 8. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 16 July 2013 21:10, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 16 July 2013 20:28, Richard Oudkerk <shibturn@gmail.com> wrote:
On 16/07/2013 6:44am, Nick Coghlan wrote:
Clarifying what constitutes an internal interface in a way that doesn't require renaming anything is a necessary prerequisite for bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes its internal implemetnation API as "import pip" rather than "import _pip" and renaming it would lead to a lot of pointless code churn). Without that concern, the topic never would have come up.
BTW, how does the use of __all__ effect things? Somewhere I got the idea that if a module uses __all__ then anything not listed is internal. I take it that is wrong?
Gah, you're right. I did consider that, then forgot to include it in the proposed text.
Rather than throwing more versions of the text at the list, I'll create a tracker issue and post a proposed patch for PEP 8.
Tracker issue: http://bugs.python.org/issue18472 When I went to write this in context, I realised it made more sense to focus on defining what a *public* interface was, and how that differed from an internal interface. WIth that approach, the natural flow of the text was: * the public interface is preferentially defined by the documentation * __all__ is the main programmatic interface * leading underscores are then mostly a helper that makes it easier to remember which is which while actually *working* on the code (but still a good practice) I also spotted a few other oddities as I went through. The obviously wrong ones I just fixed, but there were a couple relating to exceptions that seem unnecessary or ill-advised given PEP 352 and 3151, plus the one about absolute vs explicit relative imports that I started a separate thread for. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 16/07/13 20:28, Richard Oudkerk wrote:
On 16/07/2013 6:44am, Nick Coghlan wrote:
Clarifying what constitutes an internal interface in a way that doesn't require renaming anything is a necessary prerequisite for bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes its internal implemetnation API as "import pip" rather than "import _pip" and renaming it would lead to a lot of pointless code churn). Without that concern, the topic never would have come up.
BTW, how does the use of __all__ effect things? Somewhere I got the idea that if a module uses __all__ then anything not listed is internal. I take it that is wrong?
That is not how I interpret __all__. In the absence of any explicit documentation, I interpret __all__ as nothing more than a list of names which wildcard imports will bring in, without necessarily meaning that other names are private. For example, I might have a module explicitly designed for wildcard imports at the interactive interpreter: from module import * brings in the functions which I expect will be useful interactively, not necessarily the entire public API. For example, pkgutil includes classes with single-underscore methods, which I take as private. It also has a function simplegeneric, which is undocumented and not listed in __all__. In in the absence of even a comment saying "Don't use this", I take it as an oversight, not policy that simplegeneric is private. -- Steven
On Tue, 16 Jul 2013 23:19:21 +1000, Steven D'Aprano <steve@pearwood.info> wrote:
On 16/07/13 20:28, Richard Oudkerk wrote:
On 16/07/2013 6:44am, Nick Coghlan wrote:
Clarifying what constitutes an internal interface in a way that doesn't require renaming anything is a necessary prerequisite for bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes its internal implemetnation API as "import pip" rather than "import _pip" and renaming it would lead to a lot of pointless code churn). Without that concern, the topic never would have come up.
BTW, how does the use of __all__ effect things? Somewhere I got the idea that if a module uses __all__ then anything not listed is internal. I take it that is wrong?
That is not how I interpret __all__. In the absence of any explicit documentation, I interpret __all__ as nothing more than a list of names which wildcard imports will bring in, without necessarily meaning that other names are private. For example, I might have a module explicitly designed for wildcard imports at the interactive interpreter:
from module import *
brings in the functions which I expect will be useful interactively, not necessarily the entire public API.
For example, pkgutil includes classes with single-underscore methods, which I take as private. It also has a function simplegeneric, which is undocumented and not listed in __all__. In in the absence of even a comment saying "Don't use this", I take it as an oversight, not policy that simplegeneric is private.
I think you'd be wrong about that, though. simplegeneric should really be treated as private. I'm speaking here not about the general principle of the thing, but about my understanding of simplegeneric's specific history. --David
On 16 July 2013 23:39, R. David Murray <rdmurray@bitdance.com> wrote:
On Tue, 16 Jul 2013 23:19:21 +1000, Steven D'Aprano <steve@pearwood.info> wrote:
On 16/07/13 20:28, Richard Oudkerk wrote:
On 16/07/2013 6:44am, Nick Coghlan wrote:
Clarifying what constitutes an internal interface in a way that doesn't require renaming anything is a necessary prerequisite for bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes its internal implemetnation API as "import pip" rather than "import _pip" and renaming it would lead to a lot of pointless code churn). Without that concern, the topic never would have come up.
BTW, how does the use of __all__ effect things? Somewhere I got the idea that if a module uses __all__ then anything not listed is internal. I take it that is wrong?
That is not how I interpret __all__. In the absence of any explicit documentation, I interpret __all__ as nothing more than a list of names which wildcard imports will bring in, without necessarily meaning that other names are private. For example, I might have a module explicitly designed for wildcard imports at the interactive interpreter:
from module import *
brings in the functions which I expect will be useful interactively, not necessarily the entire public API.
For example, pkgutil includes classes with single-underscore methods, which I take as private. It also has a function simplegeneric, which is undocumented and not listed in __all__. In in the absence of even a comment saying "Don't use this", I take it as an oversight, not policy that simplegeneric is private.
I think you'd be wrong about that, though. simplegeneric should really be treated as private. I'm speaking here not about the general principle of the thing, but about my understanding of simplegeneric's specific history.
And, indeed, you're correct (the issue that eventually became the functools.singledispatch PEP started life with a title like "move simplegeneric to functools and make it public"). For the general case, the patch I posted to the issue tracker sets the precedence as docs -> __all__ -> leading underscore. If a module has public APIs that aren't in __all__, it should cover them in the docs, otherwise people should assume they're private. It's OK if there are exceptions to that general rule - there's a reason PEP 8 has the great big qualifier pointing out that it isn't universally applicable (even if we might sometimes wish otherwise). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 7/16/2013 9:39 AM, R. David Murray wrote:
On Tue, 16 Jul 2013 23:19:21 +1000, Steven D'Aprano <steve@pearwood.info> wrote:
For example, pkgutil includes classes with single-underscore methods, which I take as private. It also has a function simplegeneric, which is undocumented and not listed in __all__. In in the absence of even a comment saying "Don't use this", I take it as an oversight, not policy that simplegeneric is private.
I think you'd be wrong about that, though. simplegeneric should really be treated as private. I'm speaking here not about the general principle of the thing, but about my understanding of simplegeneric's specific history.
I think Steven (valid) point is "Why not, then, say it is internal either in docs or name?"-- which in this case would be in the docs. -- Terry Jan Reedy
On Tue, 16 Jul 2013 17:46:34 -0400, Terry Reedy <tjreedy@udel.edu> wrote:
On 7/16/2013 9:39 AM, R. David Murray wrote:
On Tue, 16 Jul 2013 23:19:21 +1000, Steven D'Aprano <steve@pearwood.info> wrote:
For example, pkgutil includes classes with single-underscore methods, which I take as private. It also has a function simplegeneric, which is undocumented and not listed in __all__. In in the absence of even a comment saying "Don't use this", I take it as an oversight, not policy that simplegeneric is private.
I think you'd be wrong about that, though. simplegeneric should really be treated as private. I'm speaking here not about the general principle of the thing, but about my understanding of simplegeneric's specific history.
I think Steven (valid) point is "Why not, then, say it is internal either in docs or name?"-- which in this case would be in the docs.
I don't think that's what he was saying; but yes, we should do that :) --David
https://pypi.python.org/pypi/apipkg provides a much more effective way to denote API than an _ On Wed, Jul 17, 2013 at 12:20 PM, R. David Murray <rdmurray@bitdance.com> wrote:
On Tue, 16 Jul 2013 17:46:34 -0400, Terry Reedy <tjreedy@udel.edu> wrote:
On 7/16/2013 9:39 AM, R. David Murray wrote:
On Tue, 16 Jul 2013 23:19:21 +1000, Steven D'Aprano <steve@pearwood.info> wrote:
For example, pkgutil includes classes with single-underscore methods, which I take as private. It also has a function simplegeneric, which is undocumented and not listed in __all__. In in the absence of even a comment saying "Don't use this", I take it as an oversight, not policy that simplegeneric is private.
I think you'd be wrong about that, though. simplegeneric should really be treated as private. I'm speaking here not about the general principle of the thing, but about my understanding of simplegeneric's specific history.
I think Steven (valid) point is "Why not, then, say it is internal either in docs or name?"-- which in this case would be in the docs.
I don't think that's what he was saying; but yes, we should do that :)
--David _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com
On Jul 16, 2013, at 11:28 AM, Richard Oudkerk wrote:
BTW, how does the use of __all__ effect things? Somewhere I got the idea that if a module uses __all__ then anything not listed is internal. I take it that is wrong?
Purely technically, __all__ is there to affect how from-import-* works. I personally think it's a good idea to include all your public names, and none of your non-public names, in __all__, but it's not always easy to keep up-to-date. pyflakes has the nice benefit of complaining when something is named in __all__ that doesn't exist in the module, but that's only one part of the keeping-things-up-to-date problem. -Barry
On 7/15/2013 11:11 PM, Nick Coghlan wrote:
I'll look into adding some stronger wording at the top making it clear that while PEP 8 is a useful starting point and a good default if a project doesn't have a defined style guide of it's own, it is *not* the be-all-and-end-all for Python style guides. Treating it as such as an abuse of the PEP, pure and simple.
How about (re)naming it Style Guide for Python Standard Library Code Without 'Standard Library' in the title, I can see how some people might think it meant to apply to all code. I do not think we need to worry about the above being too narrow. -- Terry Jan Reedy
On 16 July 2013 18:41, Terry Reedy <tjreedy@udel.edu> wrote:
On 7/15/2013 11:11 PM, Nick Coghlan wrote:
I'll look into adding some stronger wording at the top making it clear that while PEP 8 is a useful starting point and a good default if a project doesn't have a defined style guide of it's own, it is *not* the be-all-and-end-all for Python style guides. Treating it as such as an abuse of the PEP, pure and simple.
How about (re)naming it Style Guide for Python Standard Library Code
Without 'Standard Library' in the title, I can see how some people might think it meant to apply to all code. I do not think we need to worry about the above being too narrow.
I reread the whole thing today - it turns out we *have* tweaked it over time to account for other people using it for their projects as well. The one genuinely standard library specific element still in it (the prohibition on the use of function annotations) is explicitly qualified as only applying to the standard library and not anything else. So I think we need to continue supporting that use case. I did find it interesting that we *don't* explicitly advise against the use of "import *" for anything other than optional accelerator modules or republishing internal interfaces through a public API, even though we advice against the practice in the tutorial. Perhaps another oversight worth correcting? Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/16/2013 07:09 AM, Nick Coghlan wrote:
I did find it interesting that we *don't* explicitly advise against the use of "import *" for anything other than optional accelerator modules or republishing internal interfaces through a public API, even though we advice against the practice in the tutorial. Perhaps another oversight worth correcting?
+1. 'from foo import *' (from any source other than another module in the same package) is a code stench far worse than anything else PEP8 proscribes. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlHl2GsACgkQ+gerLs4ltQ6+1QCgmu5k6p5otzPxvzGh5mA1Ch7t 2f0AoK2a0/m144bnIwBFLLuY9l2bdWMN =878p -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On Jul 16, 2013, at 07:34 PM, Tres Seaver wrote:
On 07/16/2013 07:09 AM, Nick Coghlan wrote:
I did find it interesting that we *don't* explicitly advise against the use of "import *" for anything other than optional accelerator modules or republishing internal interfaces through a public API, even though we advice against the practice in the tutorial. Perhaps another oversight worth correcting?
+1. 'from foo import *' (from any source other than another module in the same package) is a code stench far worse than anything else PEP8 proscribes.
Maybe we should disable it for anything that isn't at the interactive prompt <0.5 wink>. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (GNU/Linux) iQIcBAEBCAAGBQJR5e5VAAoJEBJutWOnSwa/SK4P/22KvWNRQ7sjDWJ+jebgpXyD mkUeO4r905uqp5Bvedlyj4GoFhdFPQoeQEGgQeNwiMkFxCGJDKiY1XoJAgqj2zOE vUgoyLE5YSAnz8xG/Ib3ZZPXhDJY14gu/czbX/cc83Q+cQJd0+RpnhUt5jQkxAYR k6xXF7hwTHaTYlqzdTDQUtimg21AUb0lgI5XqCMGL4aac4H69JiN+3ydI3BQxqxY MXe4h4rzDxnaBnekHYCouM1jsN6CsLeIHafJR8W4Zgm9vuxZYIha+kAQd+tILFXG d/0hpvLIGP+ubNPIvS1Ay+aYsH7PiL4keCoP0pi+llMTQv+W5wDeCumLpvXjaEFj 46c7HaRaxbBqpactsG98rZOtmBwSgEwzzPSQak+sWHDCX4RFOS3qDh3WhtY0Lw7n NdEkJ0f+6bEt2ot+emDpth/SAxgOEUjNOhZ8/glYtvZcz2wBWRGkCAIEwSJtP1VZ YgRHFsoUUbS6KoI9cZWhwynmIVQf4Wn9a2Zx5/YUbHIM2mY1s60MxcOn3KgPpEU4 Zjp/gyPHsoIORrZU2Q2pwzdWEmSzorZBP/NqRqlhwBTexKdfqyRFztsFMVcnLPDv Cl0t04BGXH8/0fZ+peEOccqnuPsHumr9VqmhMCcLChRPRQyBD9ETSPVwHhprdtka qkLcyWl/M3ijyuAkdYbl =Q+Du -----END PGP SIGNATURE-----
On Tue, Jul 16, 2013 at 9:07 PM, Barry Warsaw <barry@python.org> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On Jul 16, 2013, at 07:34 PM, Tres Seaver wrote:
On 07/16/2013 07:09 AM, Nick Coghlan wrote:
I did find it interesting that we *don't* explicitly advise against the use of "import *" for anything other than optional accelerator modules or republishing internal interfaces through a public API, even though we advice against the practice in the tutorial. Perhaps another oversight worth correcting?
+1. 'from foo import *' (from any source other than another module in the same package) is a code stench far worse than anything else PEP8 proscribes.
Maybe we should disable it for anything that isn't at the interactive prompt <0.5 wink>.
Or how about dropping the whole ``from ... import ...`` format entirely? <0.5 wink; seriously, it's implementation is a pain and it has wonky semantics>
On Jul 17, 2013, at 09:35 AM, Brett Cannon wrote:
Or how about dropping the whole ``from ... import ...`` format entirely? <0.5 wink; seriously, it's implementation is a pain and it has wonky semantics>
Yes, let's break *everything* :) -Barry
On Jul 15, 2013, at 11:02 PM, Chris McDonough wrote:
Given that misunderstanding, is there a way to divorce stdlib-centric guidelines like the one being discussed now from "PEP8"-ness? I don't think any amount of marketing effort or reasoned explanation is going to separate "PEP8" from "correct code" in people's minds at this point.
The standard seems to be "if the pep8 program complains, I forward that upstream", so it's probably most effective to discuss with them. OTOH, we've had to clarify PEP 8 in order to get the pep8 developers to adjust their tool, most recently e.g. the closing brace/bracket/paren for multiline collections. -Barry
On 16 July 2013 10:23, Chris McDonough <chrism@plope.com> wrote:
On Mon, 2013-07-15 at 18:40 -0400, Barry Warsaw wrote:
Working from what I think is the latest version.
In general, i'd rather be prescriptive of future conventions than descriptive of current conventions. It's okay to exempt existing code, and as a general rule we've never been fond of rewriting existing code to update it to new standards or APIs. We don't need to do so here either.
FWIW, I'm very skeptical of a PEP 8 guideline that would try to proscribe that the "non-internal" API of a module or class would be defined solely by a naming convention.
If what's being described here does become a rule, there is reason to believe that future users who treat this PEP as the word-of-god (and there are a *lot* of them; I hear from people literally every week who want to "PEP8-ify" my code in some limited-value-added way) will be harmed. They'll be living in a fantasy world where every non-underscore-prefixed thing is now a defacto API. But I have lived in a world where that has not been the case since 1998, and the chance that I'll go back and change all my public code to satisfy a questionable introspection convention is pretty slim.
Hence the "just say it's an internal API in the docstring" escape clause. That's a pretty low bar to meet for PEP 8 compliance, and one we need, too (for things like test, idlelib and a bundled pip). I do need to figure out how to tweak the wording to make it clear that internal nature of an API applies to all contained APIs as well. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Jul 15, 2013, at 08:23 PM, Chris McDonough wrote:
On Mon, 2013-07-15 at 18:40 -0400, Barry Warsaw wrote:
Working from what I think is the latest version.
In general, i'd rather be prescriptive of future conventions than descriptive of current conventions. It's okay to exempt existing code, and as a general rule we've never been fond of rewriting existing code to update it to new standards or APIs. We don't need to do so here either.
FWIW, I'm very skeptical of a PEP 8 guideline that would try to proscribe that the "non-internal" API of a module or class would be defined solely by a naming convention.
If what's being described here does become a rule, there is reason to believe that future users who treat this PEP as the word-of-god (and there are a *lot* of them; I hear from people literally every week who want to "PEP8-ify" my code in some limited-value-added way) will be harmed. They'll be living in a fantasy world where every non-underscore-prefixed thing is now a defacto API. But I have lived in a world where that has not been the case since 1998, and the chance that I'll go back and change all my public code to satisfy a questionable introspection convention is pretty slim.
I can sympathize with this. What I tried to get at, but probably didn't succeed, is to say "if it starts with a single underscore, treat it as internal" but not necessarily the converse, i.o.w. if it *doesn't* start with an underscore that does not imply that it's public. -Barry
On 14/07/13 21:09, Nick Coghlan wrote:
Slight adjustment to the proposed wording to ensure completely undocumented modules are also considered private:
-1 on this adjustment. If somebody cannot be bothered writing a one-line doc string: "This module is private, don't touch." then they certainly shouldn't be allowed to use up a public name for a private module. I don't think we should be encouraging more private, undocumented modules. (Documentation is valuable even for private modules.) I'd go further, and say that no more private modules should be accepted for the std lib unless they have a leading underscore. I suppose for backwards compatibility reasons, we probably can't go through the std lib and rename private modules to make it clear they are private, but we don't have to accept new ones without the underscore. -- Steven
On 15Jul2013 09:48, Steven D'Aprano <steve@pearwood.info> wrote: | On 14/07/13 21:09, Nick Coghlan wrote: | | >Slight adjustment to the proposed wording to ensure completely undocumented | >modules are also considered private: | | -1 on this adjustment. If somebody cannot be bothered writing a one-line doc string: | | "This module is private, don't touch." | | then they certainly shouldn't be allowed to use up a public name for a private module. I don't think we should be encouraging more private, undocumented modules. (Documentation is valuable even for private modules.) | | I'd go further, and say that no more private modules should be accepted for the std lib unless they have a leading underscore. I suppose for backwards compatibility reasons, we probably can't go through the std lib and rename private modules to make it clear they are private, but we don't have to accept new ones without the underscore. I disagree. A private module is a perfectly sane way to implement the internals of something, especially if it is subject to implementation change in the future. Clarification: is Nick classifying a module with docstrings by no content in the "modules" section of the Python doco as undocumented? That is what I would presume; I'd expect the code to be littered with docstrings anyway, but the module as a whole is not presented in the documentation and so should be private and not relied upon. Cheers, -- Cameron Simpson <cs@zip.com.au> I sometimes wish that people would put a little more emphasis upon the observance of the law than they do upon its enforcement. - Calvin Coolidge
On Sun, Jul 14, 2013 at 8:01 PM, Cameron Simpson <cs@zip.com.au> wrote:
On 15Jul2013 09:48, Steven D'Aprano <steve@pearwood.info> wrote: | On 14/07/13 21:09, Nick Coghlan wrote: | | >Slight adjustment to the proposed wording to ensure completely undocumented | >modules are also considered private: | | -1 on this adjustment. If somebody cannot be bothered writing a one-line doc string: | | "This module is private, don't touch." | | then they certainly shouldn't be allowed to use up a public name for a private module. I don't think we should be encouraging more private, undocumented modules. (Documentation is valuable even for private modules.) | | I'd go further, and say that no more private modules should be accepted for the std lib unless they have a leading underscore. I suppose for backwards compatibility reasons, we probably can't go through the std lib and rename private modules to make it clear they are private, but we don't have to accept new ones without the underscore.
I disagree.
A private module is a perfectly sane way to implement the internals of something, especially if it is subject to implementation change in the future.
Clarification: is Nick classifying a module with docstrings by no content in the "modules" section of the Python doco as undocumented?
Yes. This has nothing to do with docstrings, just the official documentation at docs.python.org.
That is what I would presume; I'd expect the code to be littered with docstrings anyway, but the module as a whole is not presented in the documentation and so should be private and not relied upon.
Exactly.
On Mon, Jul 15, 2013 at 10:01:17AM +1000, Cameron Simpson wrote:
On 15Jul2013 09:48, Steven D'Aprano <steve@pearwood.info> wrote:
| I'd go further, and say that no more private modules should be | accepted for the std lib unless they have a leading underscore. I | suppose for backwards compatibility reasons, we probably can't go | through the std lib and rename private modules to make it clear they | are private, but we don't have to accept new ones without the | underscore.
I disagree.
A private module is a perfectly sane way to implement the internals of something, especially if it is subject to implementation change in the future.
Of course private modules are sane. I never suggested "no new private modules at all". But putting them in the same namespace as public modules is not, just to save a leading underscore in the file name. You don't even have to use the underscore in your own code: import _stuff as stuff is allowed, and doesn't make _stuff.py public since imported modules are considered implementation details by default. -- Steven
On 15 Jul 2013 13:46, "Steven D'Aprano" <steve@pearwood.info> wrote:
On Mon, Jul 15, 2013 at 10:01:17AM +1000, Cameron Simpson wrote:
On 15Jul2013 09:48, Steven D'Aprano <steve@pearwood.info> wrote:
| I'd go further, and say that no more private modules should be | accepted for the std lib unless they have a leading underscore. I | suppose for backwards compatibility reasons, we probably can't go | through the std lib and rename private modules to make it clear they | are private, but we don't have to accept new ones without the | underscore.
I disagree.
A private module is a perfectly sane way to implement the internals of something, especially if it is subject to implementation change in the future.
Of course private modules are sane. I never suggested "no new private modules at all". But putting them in the same namespace as public modules is not, just to save a leading underscore in the file name.
It's not to save a leading underscore - it's to avoid renaming existing packages like pip and idlelib. Cheers, Nick.
You don't even have to use the underscore in your own code:
import _stuff as stuff
is allowed, and doesn't make _stuff.py public since imported modules are considered implementation details by default.
-- Steven _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
Nick Coghlan writes:
Of course private modules are sane. I never suggested "no new private modules at all". But putting them in the same namespace as public modules is not, just to save a leading underscore in the file name.
It's not to save a leading underscore - it's to avoid renaming existing packages like pip and idlelib.
"existing" is not a subset of "new." Am I missing something?<wink/> What Steven is suggesting is that there are two kinds of private package in the stdlib: those that can conveniently be given names that indicate that they are private (generally, "new" packages), and those that can't ("existing" packages). For the latter IIRC he suggested adding a line at the top of the module documentation "This module is *private*, and you cannot depend on API compatibility with future or past releases of this module. The public API is in ... [alternatively, There is no public API]." I really don't see how one can object to this suggestion, given an appropriate definition of "convenient enough to get a _name".
On 15 July 2013 09:48, Steven D'Aprano <steve@pearwood.info> wrote:
On 14/07/13 21:09, Nick Coghlan wrote:
Slight adjustment to the proposed wording to ensure completely undocumented modules are also considered private:
-1 on this adjustment. If somebody cannot be bothered writing a one-line doc string:
"This module is private, don't touch."
then they certainly shouldn't be allowed to use up a public name for a private module. I don't think we should be encouraging more private, undocumented modules. (Documentation is valuable even for private modules.)
For context, this arose when I checked PEP 8 after the point was raised on distutils-sig that pip uses a public name for its implementation module (also pip, same as the CLI), but officially exposes no stable public programmatic API. At the moment, according to what little PEP 8 has to say about public vs private interfaces, that means a bundled pip would represent a new public API, despite the fact that the only documented interface for pip is the CLI, not the Python module API. Since we've been burned by people assuming private APIs are public in the past, I figured it made sense to get the actual standards we follow in practice documented properly, *before* anyone further gets the wrong idea from "import pip; help(pip)". Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Mon, Jul 15, 2013 at 10:30:02AM +1000, Nick Coghlan wrote:
On 15 July 2013 09:48, Steven D'Aprano <steve@pearwood.info> wrote:
On 14/07/13 21:09, Nick Coghlan wrote:
Slight adjustment to the proposed wording to ensure completely undocumented modules are also considered private:
-1 on this adjustment. If somebody cannot be bothered writing a one-line doc string:
"This module is private, don't touch."
then they certainly shouldn't be allowed to use up a public name for a private module. I don't think we should be encouraging more private, undocumented modules. (Documentation is valuable even for private modules.)
For context, this arose when I checked PEP 8 after the point was raised on distutils-sig that pip uses a public name for its implementation module (also pip, same as the CLI), but officially exposes no stable public programmatic API.
I don't think we should bless this as official policy, certainly not for the standard library. We can't tell external modules what to do, and possibly there's nothing we can do about existing undocumented private modules, short of renaming them, but I think that at least for new additions to the std lib module names should follow the same single-underscore convention as functions, classes etc. A single leading underscore introduces the private namespace, everything else is public. I've seen too much undocumented public code to ever assume that lack of documentation implies it is private.
At the moment, according to what little PEP 8 has to say about public vs private interfaces, that means a bundled pip would represent a new public API, despite the fact that the only documented interface for pip is the CLI, not the Python module API.
Since we've been burned by people assuming private APIs are public in the past, I figured it made sense to get the actual standards we follow in practice documented properly, *before* anyone further gets the wrong idea from "import pip; help(pip)".
I can tell you that people will do that regardless of what PEP 8 has to say about "not documented == private". Especially as worded. I'm sure I won't be the only person to reason that if a module has a docstring, it is therefore documented. It takes one character to explicitly document a module as private. That's much better than expecting people to treat their failure to find documentation as an implicit warning. -- Steven
On 15 Jul 2013 13:43, "Steven D'Aprano" <steve@pearwood.info> wrote:
On Mon, Jul 15, 2013 at 10:30:02AM +1000, Nick Coghlan wrote:
On 15 July 2013 09:48, Steven D'Aprano <steve@pearwood.info> wrote:
On 14/07/13 21:09, Nick Coghlan wrote:
Slight adjustment to the proposed wording to ensure completely undocumented modules are also considered private:
-1 on this adjustment. If somebody cannot be bothered writing a
one-line doc
string:
"This module is private, don't touch."
then they certainly shouldn't be allowed to use up a public name for a private module. I don't think we should be encouraging more private, undocumented modules. (Documentation is valuable even for private modules.)
For context, this arose when I checked PEP 8 after the point was raised on distutils-sig that pip uses a public name for its implementation module (also pip, same as the CLI), but officially exposes no stable public programmatic API.
I don't think we should bless this as official policy, certainly not for the standard library. We can't tell external modules what to do, and possibly there's nothing we can do about existing undocumented private modules, short of renaming them, but I think that at least for new additions to the std lib module names should follow the same single-underscore convention as functions, classes etc. A single leading underscore introduces the private namespace, everything else is public.
I've seen too much undocumented public code to ever assume that lack of documentation implies it is private.
At the moment, according to what little PEP 8 has to say about public vs private interfaces, that means a bundled pip would represent a new public API, despite the fact that the only documented interface for pip is the CLI, not the Python module API.
Since we've been burned by people assuming private APIs are public in the past, I figured it made sense to get the actual standards we follow in practice documented properly, *before* anyone further gets the wrong idea from "import pip; help(pip)".
I can tell you that people will do that regardless of what PEP 8 has to say about "not documented == private". Especially as worded. I'm sure I won't be the only person to reason that if a module has a docstring, it is therefore documented.
It takes one character to explicitly document a module as private. That's much better than expecting people to treat their failure to find documentation as an implicit warning.
We're considering bundling the pip CLI for the convenience of educators and beginners, not adding their implementation module to the standard library. That means we need a bright shining line for what constitutes "in the standard library". Yes, it's better if people properly use a leading underscore, but the fact is many don't (including idlelib, as Terry noted) and it becomes a disruptive change to add it later if people don't start with it. The pip and IDLE developers have better things to be doing than changing the names of undocumented APIs just because the application is distributed along with CPython. Cheers, Nick.
-- Steven _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
On 7/14/2013 7:09 AM, Nick Coghlan wrote:
Slight adjustment to the proposed wording to ensure completely undocumented modules are also considered private:
================= Private interfaces
Unless explicitly documented otherwise, a leading underscore on any name indicates that it is an internal implementation detail and backwards compatibility guarantees do not apply. It is strongly encouraged that private APIs (whether modules, classes, functions, attributes or other names) be clearly marked in this way, as Python users may rely on introspection to identify available functionality and may be misled into believing an API without a leading underscore is in fact a public API with the standard backwards compatibility guarantees.
Even when their names do not start with a leading underscore, all test modules and all modules that are not covered in the documentation are also considered private interfaces.
I was going to suggest adding 'and most idlelib ' between 'test' and 'modules', but the broader addition covers idlelib, which is not mentioned yet in the docs, even in the (unindexed) Idle chapter. When it is, I will try to remember to make explicit which names and interfaces are public and that the rest are private. -- Terry Jan Reedy
On Jul 14, 2013, at 06:11 PM, Nick Coghlan wrote:
Private interfaces
PEP 8 does say: _single_leading_underscore: weak "internal use" indicator. E.g. from M import * does not import objects whose name starts with an underscore. I'm in favor of making this a stronger recommendation, but I have a small semantic quibble. Instead of "private interface" it should be "non-public interface". The two aren't quite the same thing; for example, often single-leading underscores are used for methods that subclasses are supposed to override, e.g. akin to "protected" in C++. Besides, Python doesn't really have any strong notion of privateness, so saying "non-public" means "just because you can, doesn't mean you should". -Barry
On Mon, 15 Jul 2013 15:51:31 -0400 Barry Warsaw <barry@python.org> wrote:
On Jul 14, 2013, at 06:11 PM, Nick Coghlan wrote:
Private interfaces
PEP 8 does say:
_single_leading_underscore: weak "internal use" indicator. E.g. from M import * does not import objects whose name starts with an underscore.
I'm in favor of making this a stronger recommendation, but I have a small semantic quibble. Instead of "private interface" it should be "non-public interface". The two aren't quite the same thing; for example, often single-leading underscores are used for methods that subclasses are supposed to override, e.g. akin to "protected" in C++.
C++ doesn't have a monopoly over the definition of "private".
Besides, Python doesn't really have any strong notion of privateness, so saying "non-public" means "just because you can, doesn't mean you should".
If it doesn't have any notion of privateness, then it can't have a notion of "publicness" either. If you really want another word (I am personally fine with "private"), "internal" it should be. Regards Antoine.
On Jul 15, 2013, at 09:56 PM, Antoine Pitrou wrote:
If you really want another word (I am personally fine with "private"), "internal" it should be.
I would be fine with "internal", since that's how PEP 8 already classifies such names. :) -Barry
On 16 Jul 2013 06:13, "Barry Warsaw" <barry@python.org> wrote:
On Jul 15, 2013, at 09:56 PM, Antoine Pitrou wrote:
If you really want another word (I am personally fine with "private"), "internal" it should be.
I would be fine with "internal", since that's how PEP 8 already classifies such names. :)
Yeah, internal is a better word for this. Cheers, Nick.
-Barry _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
participants (15)
-
Antoine Pitrou -
Barry Warsaw -
Brett Cannon -
Cameron Simpson -
Chris McDonough -
Daniel Holth -
Gregory P. Smith -
Lennart Regebro -
Nick Coghlan -
R. David Murray -
Richard Oudkerk -
Stephen J. Turnbull -
Steven D'Aprano -
Terry Reedy -
Tres Seaver