On Sun, Jul 14, 2013 at 7:09 AM, Nick Coghlan wrote:

On 14 July 2013 18:11, Nick Coghlan wrote:

...

Currently, the naming section of PEP 8 doesn't say very much about what a leading underscore *means* in the Python standard library.

I would like to add a new "Private interfaces" subsection under "Naming Conventions" to say the following:

================= Private interfaces

Unless explicitly documented otherwise, a leading underscore on any name indicates that it is an internal implementation detail and backwards compatibility guarantees do not apply. It is strongly encouraged that private APIs (whether modules, classes, functions, attributes or other names) be clearly marked in this way, as many Python users rely on introspection to identify available functionality and may be mislead into believing an API without the leading underscore is in fact a public API with the standard backwards compatibility guarantees.

All test modules are also considered private interfaces.

Even though they typically lack the leading underscore, modules imported by another module are also considered an implementation detail. Other modules *should* not rely on indirect access to such modules unless they are an explicitly documented part of the API (such as ``os.path``). =================

Slight adjustment to the proposed wording to ensure completely undocumented modules are also considered private:

================= Private interfaces

Unless explicitly documented otherwise, a leading underscore on any name indicates that it is an internal implementation detail and backwards compatibility guarantees do not apply. It is strongly encouraged that private APIs (whether modules, classes, functions, attributes or other names) be clearly marked in this way, as Python users may rely on introspection to identify available functionality and may be misled into believing an API without a leading underscore is in fact a public API with the standard backwards compatibility guarantees.

Even when their names do not start with a leading underscore, all test modules and all modules that are not covered in the documentation are also considered private interfaces.

Similarly, the specific modules and external APIs imported by a module are always considered an implementation detail. Other modules should not rely on indirect access to such imported interfaces unless they are an explicitly documented part of the containing module's API (such as ``os.path`` or a package ``__init__`` module exposing functionality from submodules). =================

+1

On 15 July 2013 02:15, Gregory P. Smith wrote:

On Sun, Jul 14, 2013 at 4:09 AM, Nick Coghlan wrote:

...

as Python users may rely on introspection to identify available functionality and may be misled into believing an API without a leading underscore is in fact a public API with the standard backwards compatibility guarantees.

While true, I'm not sure the last part of the sentence is necessary. Once we've established that a leading _ indicates something is private there isn't much point in explaining the various ways people might find them. I'm happy regardless of this bit being there.

You'd be surprised how many non-core devs react with astonishment when I suggest that not documenting something isn't enough to avoid having users consider it a supported public API - they usually get it after I point out how far you can usually get just by using dir() and help() to play with a new library at the interactive prompt instead of looking at out-of-band docs. I figure including the second part will help prevent some "But why?" reactions in the future. I think Steven has a reasonable point about being clearer that an explicit leading underscore is the preferred solution for any new private modules, though, so here's an updated proposal: ================= Private interfaces Unless explicitly documented otherwise, a leading underscore on any name indicates that it is an internal implementation detail and any backwards compatibility guarantees do not apply. It is strongly encouraged that private APIs (whether modules, classes, functions, attributes or other names) be clearly marked in this way, as Python users may rely on introspection to identify available functionality and may be misled into believing an API without a leading underscore is in fact a public API with the standard backwards compatibility guarantees. While the explicit use of a leading underscore in the names of private modules is preferred, all test modules and all modules that are not explicitly covered in the documentation are also considered private interfaces, even when their names do not start with a leading underscore and even if they include a module level documentation string. This includes submodules of packages that are documented as if they were a single module. Similarly, the specific modules and external APIs imported by a module are always considered an implementation detail. Other modules should not rely on indirect access to such imported interfaces unless they are an explicitly documented part of the containing module's API (such as ``os.path`` or a package ``__init__`` module that exposes functionality from submodules). ================= -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 7/15/2013 12:17 AM, Nick Coghlan wrote:

You'd be surprised how many non-core devs react with astonishment when I suggest that not documenting something isn't enough to avoid having users consider it a supported public API - they usually get it after I point out how far you can usually get just by using dir() and help() to play with a new library at the interactive prompt instead of looking at out-of-band docs. I figure including the second part will help prevent some "But why?" reactions in the future.

I agree with the goal of preventing such reactions, but I suggest slightly different means to do so. I just looked and idlelib.__init__.py consists of """# Dummy file to make this a package. """ help(idlelib) prints the comment (which surprised me, but it seems to be smart enough to look for a 'doc comment' when it does not find a doc string). I now plan to change the comment to a docstring so 'idlelib.__doc__' also works and change the text to something like """A package of private modules that implement the IDLE integrated shell and editor application. See the manual for the public entry points.""" (or maybe list them briefly here in the doc string). This should properly inform people who introspect. Also, the manual needs a small, indexed, section on idlelib that says much the same thing along with documenting what *is* public (which is not completely clear to me at the moment).

I think Steven has a reasonable point about being clearer that an explicit leading underscore is the preferred solution for any new private modules, though, so here's an updated proposal:

The public/private distinction is not always as clean-cut as seems to be assumed in this discussion. Test and idlelib both have public entry points to an application based on private files. ('Running the Python test suite' is a batch application.) So I do not think either should be _ prefixed. For both, and anything similar, a proper docstring and manual entry explaining the nuances should be completely acceptable as a substitute for a blunt, all-or-nothing flag. I also think that marking something private, either way, or partially private with words, should extend to its contents. In particular, the 100s of implementation files in test and idlelib need not be _ marked as as they would be covered once test and idlelib are properly documented. The special _ mark makes sense when it is the exception, but not in a context where it would be the overwhelming norm, because everything but one or two files is private.

================= Private interfaces

Unless explicitly documented otherwise, a leading underscore on any name indicates that it is an internal implementation detail and any backwards compatibility guarantees do not apply. It is strongly encouraged that private APIs (whether modules, classes, functions, attributes or other names) be clearly marked in this way, as Python users may rely on introspection to identify available functionality and may be misled into believing an API without a leading underscore is in fact a public API with the standard backwards compatibility guarantees.

While the explicit use of a leading underscore in the names of private modules is preferred, all test modules and all modules that are not explicitly covered in the documentation are also considered private interfaces, even when their names do not start with a leading underscore and even if they include a module level documentation string. This includes submodules of packages that are documented as if they were a single module.

Since I am planning to explicitly cover idlelib in the doc, to list the public interface, just as is done for test, I would like it explicitly mentioned along with test. I do not think absence of documentation is a good signal. I would rather say that any private or partially private package that does not have a _ name *must* have a docstring saying that it is all or mostly private. (Or be a subpackage or module of something so marked.) And presuming that it is associated with something that is documented, then the doc should state that the implementation files are private. In other words, I suggest that "modules that are not explicitly covered in the documentation ... [whose] names do not start with a leading underscore" should be empty. And I think "even if they include a module level documentation string." is backwards. Such files *should* have a docstring that serves as a substitute for a _ prefix.

Similarly, the specific modules and external APIs imported by a module are always considered an implementation detail. Other modules should not rely on indirect access to such imported interfaces unless they are an explicitly documented part of the containing module's API (such as ``os.path`` or a package ``__init__`` module that exposes functionality from submodules). =================

-- Terry Jan Reedy

Working from what I think is the latest version. In general, i'd rather be prescriptive of future conventions than descriptive of current conventions. It's okay to exempt existing code, and as a general rule we've never been fond of rewriting existing code to update it to new standards or APIs. We don't need to do so here either. On Jul 15, 2013, at 05:48 PM, Nick Coghlan wrote:

Private interfaces

"Internal" or "Non-public"

Unless explicitly documented otherwise, a leading underscore on any name indicates that it is an internal implementation detail and any backwards compatibility guarantees do not apply. It is strongly encouraged that private APIs (whether modules, classes, functions, attributes or other names) be clearly marked in this way, as Python users may rely on introspection to identify available functionality and may be misled into believing an API without a leading underscore is in fact a public API with the standard backwards compatibility guarantees.

How about: "All internal interfaces (modules, classes, functions, attributes or other names) should be prefixed with a single leading underscore. Such names are internal implementation details for which no backward compatibility guarantees are made, unless otherwise specified. Existing code and other narrowly accepted exceptions may override this recommendation, in which case the docstrings and/or documentation for such code must clearly and explicitly state the internal status of the APIs. Imported names should always be considered an implementation detail. Other modules must not rely on indirect access to such imported names unless they are an explicitly documented part of the containing module's API, such as ``os.path`` or a package's ``__init__`` module that exposes functionality from submodules. Public names exported by a module should be include in the module's ``__all__`` attribute."

While the explicit use of a leading underscore is the preferred solution, the names of some private (or partially private) modules (such as ``test`` and ``idlelib``) lack the leading underscore either for historical reasons or because they expose a public command line interface through the ``-m`` switch. Such modules should include an explicit disclaimer in their module docstring, indicating that they do not use the leading underscore convention and noting where the definition of the public API (if any) can be found (the public API definition may also be part of the module docstring).

As a general principle, the specific modules and external APIs imported by a module are always considered an implementation detail. Other modules should not rely on indirect access to such imported interfaces unless they are an explicitly documented part of the containing module's API (such as ``os.path`` or a package ``__init__`` module that exposes functionality from submodules).

Cheers, -Barry

On 16/07/13 10:23, Chris McDonough wrote:

If what's being described here does become a rule, there is reason to believe that future users who treat this PEP as the word-of-god (and there are a *lot* of them; I hear from people literally every week who want to "PEP8-ify" my code in some limited-value-added way) will be harmed.

I sympathise with your pain, but is that not true of every PEP 8 naming convention?

They'll be living in a fantasy world where every non-underscore-prefixed thing is now a defacto API.

If your code has no obvious, documented convention at all for what's internal and what is not, they are no worse off. If you do have a documented convention for internal implementation details, then you are no worse off. "I have better things to do than PEP8-ify old, working, stable code" is a perfectly acceptable answer. "I have better things to do than PEP9-ify old, working, stable code, but if you want to provide regression tests and a working patch, I'll let you do so" might be an even better one :-) -- Steven

On 16 July 2013 12:20, Chris McDonough wrote:

On Tue, 2013-07-16 at 11:25 +1000, Steven D'Aprano wrote:

...

If your code has no obvious, documented convention at all for what's internal and what is not, they are no worse off.

If you do have a documented convention for internal implementation details, then you are no worse off. "I have better things to do than PEP8-ify old, working, stable code" is a perfectly acceptable answer. "I have better things to do than PEP9-ify old, working, stable code, but if you want to provide regression tests and a working patch, I'll let you do so" might be an even better one :-)

Welp, I guess I'm logically boxed in then. Thanks for showing me the errors in my thinking. Should be no problem to manage the updating of that 500K lines of public code.

/scarcasm

How do get from "If this doesn't apply to a module, just add something like 'This is an internal API' or 'This module includes internal APIs, consult the documentation for the public API' to the module docstring" to "updating 500k lines of public code"? The version in Barry's email that you replied to has that escape clause in it, so the fact it was missing from my original text doesn't justify this reaction. Cheers, Nick. P.S. Note that, while I'm trying to account for it in this particular case, we're never going to let the fact that many people misuse PEP 8 by considering it as a holy standard that should be followed by all Python code everywhere stop us from including updates that are valid specifically for the standard library. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 16 July 2013 13:02, Chris McDonough wrote:

OSS developers have spent many months jumping through bw incompat hoops in Python over the last few years, and it has taken time away from doing things that provide value. The less I can do of that, the better, and Python gets more value too. That said, I realize that I'm in the minority because I happen to have a metric ton of public code out there. But it'd be nice if that was encouraged rather than effectively punished on the hunch that it might provide some benefit for a theoretical new user.

You, Armin and everyone else that works on the bytes/text boundary are indeed the hardest hit by the Python 3 transition, and I appreciate the hard work you have all done to help make that transition as successful as it has been so far. However, the fact that people abuse PEP 8 by treating it as "all Python code in the world should follow these rules" cannot, and will not, stop us from continuing to use it to set appropriate guidelines *for the standard library*. I'll look into adding some stronger wording at the top making it clear that while PEP 8 is a useful starting point and a good default if a project doesn't have a defined style guide of it's own, it is *not* the be-all-and-end-all for Python style guides. Treating it as such as an abuse of the PEP, pure and simple. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 16 July 2013 13:16, Chris McDonough wrote:

I understand that. Unfortunately the remainder of the world does not. The same IDEs that would be helped via this proposed change have "PEP8 modes" turned on *by default*! http://blog.jetbrains.com/pycharm/2013/02/long-awaited-pep-8-checks-on-the-f...

It seems like an unwise step to continue stuffing things into the PEP8 brand bag as a result if that stuff is only meant to apply to the stdlib.

Clarifying what constitutes an internal interface in a way that doesn't require renaming anything is a necessary prerequisite for bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes its internal implemetnation API as "import pip" rather than "import _pip" and renaming it would lead to a lot of pointless code churn). Without that concern, the topic never would have come up. Yes, this new section may give the PEP 8 prescriptivists more ammunition to complain about existing projects with different coding standards not complying with the latest standard library guidelines. We're not unfamiliar with the complaint, given the number of standard library modules (some dating from the 20th century) that don't comply with the current incarnation of PEP 8 circa 2013. However, PEP 8 is already quite explicit about what we think of that kind of behaviour: http://www.python.org/dev/peps/pep-0008/#a-foolish-consistency-is-the-hobgob... PEP 8 is a tool. Like many of the other tools we provide as part of Python, it can be abused. However, the likelihood of such abuse is *not* an excuse to avoid using the tool appropriately, it's a reason to continue the thankless task of trying to educate people on what's appropriate and what isn't (and making wholesale changes to working code just to comply with a style guide generally isn't appropriate). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 16 July 2013 16:01, Chris McDonough wrote:

Well, I know it's arguing with the umpire at this point, but PEP8 prescriptionism (great word by the way!) is de rigeur these days. It's a python-dev perogative to ignore that, but it has consequences that reverberate further than this maillist.

Yeah, we should probably be a bit more vocal in pointing out that PEP 8 is something to be applied judiciously, *not* treated as "sacred writ, never to be questioned". It's very tempting to just sigh and ignore it, instead of explicitly pointing out the problems with taking it too literally for people's personal projects (or, worse, using it to abuse developers of projects that long predate the recommendations in more recent versions) :P Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 16/07/2013 6:44am, Nick Coghlan wrote:

Clarifying what constitutes an internal interface in a way that doesn't require renaming anything is a necessary prerequisite for bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes its internal implemetnation API as "import pip" rather than "import _pip" and renaming it would lead to a lot of pointless code churn). Without that concern, the topic never would have come up.

BTW, how does the use of __all__ effect things? Somewhere I got the idea that if a module uses __all__ then anything not listed is internal. I take it that is wrong? -- Richard

On 16 July 2013 21:10, Nick Coghlan wrote:

On 16 July 2013 20:28, Richard Oudkerk wrote:

...

On 16/07/2013 6:44am, Nick Coghlan wrote:

...

Clarifying what constitutes an internal interface in a way that doesn't require renaming anything is a necessary prerequisite for bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes its internal implemetnation API as "import pip" rather than "import _pip" and renaming it would lead to a lot of pointless code churn). Without that concern, the topic never would have come up.

BTW, how does the use of __all__ effect things? Somewhere I got the idea that if a module uses __all__ then anything not listed is internal. I take it that is wrong?

Gah, you're right. I did consider that, then forgot to include it in the proposed text.

Rather than throwing more versions of the text at the list, I'll create a tracker issue and post a proposed patch for PEP 8.

Tracker issue: http://bugs.python.org/issue18472 When I went to write this in context, I realised it made more sense to focus on defining what a *public* interface was, and how that differed from an internal interface. WIth that approach, the natural flow of the text was: * the public interface is preferentially defined by the documentation * __all__ is the main programmatic interface * leading underscores are then mostly a helper that makes it easier to remember which is which while actually *working* on the code (but still a good practice) I also spotted a few other oddities as I went through. The obviously wrong ones I just fixed, but there were a couple relating to exceptions that seem unnecessary or ill-advised given PEP 352 and 3151, plus the one about absolute vs explicit relative imports that I started a separate thread for. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Tue, 16 Jul 2013 23:19:21 +1000, Steven D'Aprano wrote:

On 16/07/13 20:28, Richard Oudkerk wrote:

...

On 16/07/2013 6:44am, Nick Coghlan wrote:

...

Clarifying what constitutes an internal interface in a way that doesn't require renaming anything is a necessary prerequisite for bundling or bootstrapping the pip CLI in Python 3.4 (as pip exposes its internal implemetnation API as "import pip" rather than "import _pip" and renaming it would lead to a lot of pointless code churn). Without that concern, the topic never would have come up.

BTW, how does the use of __all__ effect things? Somewhere I got the idea that if a module uses __all__ then anything not listed is internal. I take it that is wrong?

That is not how I interpret __all__. In the absence of any explicit documentation, I interpret __all__ as nothing more than a list of names which wildcard imports will bring in, without necessarily meaning that other names are private. For example, I might have a module explicitly designed for wildcard imports at the interactive interpreter:

from module import *

brings in the functions which I expect will be useful interactively, not necessarily the entire public API.

For example, pkgutil includes classes with single-underscore methods, which I take as private. It also has a function simplegeneric, which is undocumented and not listed in __all__. In in the absence of even a comment saying "Don't use this", I take it as an oversight, not policy that simplegeneric is private.

I think you'd be wrong about that, though. simplegeneric should really be treated as private. I'm speaking here not about the general principle of the thing, but about my understanding of simplegeneric's specific history. --David

On Jul 16, 2013, at 11:48 PM, Nick Coghlan wrote:

For the general case, the patch I posted to the issue tracker sets the precedence as docs -> __all__ -> leading underscore.

+1 -Barry

On Tue, 16 Jul 2013 17:46:34 -0400, Terry Reedy wrote:

On 7/16/2013 9:39 AM, R. David Murray wrote:

...

On Tue, 16 Jul 2013 23:19:21 +1000, Steven D'Aprano wrote:

...

...

For example, pkgutil includes classes with single-underscore methods, which I take as private. It also has a function simplegeneric, which is undocumented and not listed in __all__. In in the absence of even a comment saying "Don't use this", I take it as an oversight, not policy that simplegeneric is private.

I think you'd be wrong about that, though. simplegeneric should really be treated as private. I'm speaking here not about the general principle of the thing, but about my understanding of simplegeneric's specific history.

I think Steven (valid) point is "Why not, then, say it is internal either in docs or name?"-- which in this case would be in the docs.

I don't think that's what he was saying; but yes, we should do that :) --David

On Jul 16, 2013, at 11:28 AM, Richard Oudkerk wrote:

BTW, how does the use of __all__ effect things? Somewhere I got the idea that if a module uses __all__ then anything not listed is internal. I take it that is wrong?

Purely technically, __all__ is there to affect how from-import-* works. I personally think it's a good idea to include all your public names, and none of your non-public names, in __all__, but it's not always easy to keep up-to-date. pyflakes has the nice benefit of complaining when something is named in __all__ that doesn't exist in the module, but that's only one part of the keeping-things-up-to-date problem. -Barry

On 16 July 2013 18:41, Terry Reedy wrote:

On 7/15/2013 11:11 PM, Nick Coghlan wrote:

...

I'll look into adding some stronger wording at the top making it clear that while PEP 8 is a useful starting point and a good default if a project doesn't have a defined style guide of it's own, it is *not* the be-all-and-end-all for Python style guides. Treating it as such as an abuse of the PEP, pure and simple.

How about (re)naming it Style Guide for Python Standard Library Code

Without 'Standard Library' in the title, I can see how some people might think it meant to apply to all code. I do not think we need to worry about the above being too narrow.

I reread the whole thing today - it turns out we *have* tweaked it over time to account for other people using it for their projects as well. The one genuinely standard library specific element still in it (the prohibition on the use of function annotations) is explicitly qualified as only applying to the standard library and not anything else. So I think we need to continue supporting that use case. I did find it interesting that we *don't* explicitly advise against the use of "import *" for anything other than optional accelerator modules or republishing internal interfaces through a public API, even though we advice against the practice in the tutorial. Perhaps another oversight worth correcting? Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On Jul 16, 2013, at 07:34 PM, Tres Seaver wrote:

On 07/16/2013 07:09 AM, Nick Coghlan wrote:

...

I did find it interesting that we *don't* explicitly advise against the use of "import *" for anything other than optional accelerator modules or republishing internal interfaces through a public API, even though we advice against the practice in the tutorial. Perhaps another oversight worth correcting?

+1. 'from foo import *' (from any source other than another module in the same package) is a code stench far worse than anything else PEP8 proscribes.

Maybe we should disable it for anything that isn't at the interactive prompt <0.5 wink>. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (GNU/Linux) iQIcBAEBCAAGBQJR5e5VAAoJEBJutWOnSwa/SK4P/22KvWNRQ7sjDWJ+jebgpXyD mkUeO4r905uqp5Bvedlyj4GoFhdFPQoeQEGgQeNwiMkFxCGJDKiY1XoJAgqj2zOE vUgoyLE5YSAnz8xG/Ib3ZZPXhDJY14gu/czbX/cc83Q+cQJd0+RpnhUt5jQkxAYR k6xXF7hwTHaTYlqzdTDQUtimg21AUb0lgI5XqCMGL4aac4H69JiN+3ydI3BQxqxY MXe4h4rzDxnaBnekHYCouM1jsN6CsLeIHafJR8W4Zgm9vuxZYIha+kAQd+tILFXG d/0hpvLIGP+ubNPIvS1Ay+aYsH7PiL4keCoP0pi+llMTQv+W5wDeCumLpvXjaEFj 46c7HaRaxbBqpactsG98rZOtmBwSgEwzzPSQak+sWHDCX4RFOS3qDh3WhtY0Lw7n NdEkJ0f+6bEt2ot+emDpth/SAxgOEUjNOhZ8/glYtvZcz2wBWRGkCAIEwSJtP1VZ YgRHFsoUUbS6KoI9cZWhwynmIVQf4Wn9a2Zx5/YUbHIM2mY1s60MxcOn3KgPpEU4 Zjp/gyPHsoIORrZU2Q2pwzdWEmSzorZBP/NqRqlhwBTexKdfqyRFztsFMVcnLPDv Cl0t04BGXH8/0fZ+peEOccqnuPsHumr9VqmhMCcLChRPRQyBD9ETSPVwHhprdtka qkLcyWl/M3ijyuAkdYbl =Q+Du -----END PGP SIGNATURE-----

On Jul 17, 2013, at 09:35 AM, Brett Cannon wrote:

Or how about dropping the whole ``from ... import ...`` format entirely? <0.5 wink; seriously, it's implementation is a pain and it has wonky semantics>

Yes, let's break *everything* :) -Barry

On 16 July 2013 10:23, Chris McDonough wrote:

On Mon, 2013-07-15 at 18:40 -0400, Barry Warsaw wrote:

...

Working from what I think is the latest version.

In general, i'd rather be prescriptive of future conventions than descriptive of current conventions. It's okay to exempt existing code, and as a general rule we've never been fond of rewriting existing code to update it to new standards or APIs. We don't need to do so here either.

FWIW, I'm very skeptical of a PEP 8 guideline that would try to proscribe that the "non-internal" API of a module or class would be defined solely by a naming convention.

If what's being described here does become a rule, there is reason to believe that future users who treat this PEP as the word-of-god (and there are a *lot* of them; I hear from people literally every week who want to "PEP8-ify" my code in some limited-value-added way) will be harmed. They'll be living in a fantasy world where every non-underscore-prefixed thing is now a defacto API. But I have lived in a world where that has not been the case since 1998, and the chance that I'll go back and change all my public code to satisfy a questionable introspection convention is pretty slim.

Hence the "just say it's an internal API in the docstring" escape clause. That's a pretty low bar to meet for PEP 8 compliance, and one we need, too (for things like test, idlelib and a bundled pip). I do need to figure out how to tweak the wording to make it clear that internal nature of an API applies to all contained APIs as well. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 14/07/13 21:09, Nick Coghlan wrote:

Slight adjustment to the proposed wording to ensure completely undocumented modules are also considered private:

-1 on this adjustment. If somebody cannot be bothered writing a one-line doc string: "This module is private, don't touch." then they certainly shouldn't be allowed to use up a public name for a private module. I don't think we should be encouraging more private, undocumented modules. (Documentation is valuable even for private modules.) I'd go further, and say that no more private modules should be accepted for the std lib unless they have a leading underscore. I suppose for backwards compatibility reasons, we probably can't go through the std lib and rename private modules to make it clear they are private, but we don't have to accept new ones without the underscore. -- Steven

On Sun, Jul 14, 2013 at 8:01 PM, Cameron Simpson wrote:

On 15Jul2013 09:48, Steven D'Aprano wrote: | On 14/07/13 21:09, Nick Coghlan wrote: | | >Slight adjustment to the proposed wording to ensure completely undocumented | >modules are also considered private: | | -1 on this adjustment. If somebody cannot be bothered writing a one-line doc string: | | "This module is private, don't touch." | | then they certainly shouldn't be allowed to use up a public name for a private module. I don't think we should be encouraging more private, undocumented modules. (Documentation is valuable even for private modules.) | | I'd go further, and say that no more private modules should be accepted for the std lib unless they have a leading underscore. I suppose for backwards compatibility reasons, we probably can't go through the std lib and rename private modules to make it clear they are private, but we don't have to accept new ones without the underscore.

I disagree.

A private module is a perfectly sane way to implement the internals of something, especially if it is subject to implementation change in the future.

Clarification: is Nick classifying a module with docstrings by no content in the "modules" section of the Python doco as undocumented?

Yes. This has nothing to do with docstrings, just the official documentation at docs.python.org.

That is what I would presume; I'd expect the code to be littered with docstrings anyway, but the module as a whole is not presented in the documentation and so should be private and not relied upon.

Exactly.

On 15 Jul 2013 13:46, "Steven D'Aprano" wrote:

On Mon, Jul 15, 2013 at 10:01:17AM +1000, Cameron Simpson wrote:

...

On 15Jul2013 09:48, Steven D'Aprano wrote:

...

| I'd go further, and say that no more private modules should be | accepted for the std lib unless they have a leading underscore. I | suppose for backwards compatibility reasons, we probably can't go | through the std lib and rename private modules to make it clear they | are private, but we don't have to accept new ones without the | underscore.

I disagree.

A private module is a perfectly sane way to implement the internals of something, especially if it is subject to implementation change in the future.

Of course private modules are sane. I never suggested "no new private modules at all". But putting them in the same namespace as public modules is not, just to save a leading underscore in the file name.

It's not to save a leading underscore - it's to avoid renaming existing packages like pip and idlelib. Cheers, Nick.

You don't even have to use the underscore in your own code:

import _stuff as stuff

is allowed, and doesn't make _stuff.py public since imported modules are considered implementation details by default.

-- Steven _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

On 15 July 2013 09:48, Steven D'Aprano wrote:

On 14/07/13 21:09, Nick Coghlan wrote:

...

Slight adjustment to the proposed wording to ensure completely undocumented modules are also considered private:

-1 on this adjustment. If somebody cannot be bothered writing a one-line doc string:

"This module is private, don't touch."

then they certainly shouldn't be allowed to use up a public name for a private module. I don't think we should be encouraging more private, undocumented modules. (Documentation is valuable even for private modules.)

For context, this arose when I checked PEP 8 after the point was raised on distutils-sig that pip uses a public name for its implementation module (also pip, same as the CLI), but officially exposes no stable public programmatic API. At the moment, according to what little PEP 8 has to say about public vs private interfaces, that means a bundled pip would represent a new public API, despite the fact that the only documented interface for pip is the CLI, not the Python module API. Since we've been burned by people assuming private APIs are public in the past, I figured it made sense to get the actual standards we follow in practice documented properly, *before* anyone further gets the wrong idea from "import pip; help(pip)". Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 15 Jul 2013 13:43, "Steven D'Aprano" wrote:

On Mon, Jul 15, 2013 at 10:30:02AM +1000, Nick Coghlan wrote:

...

On 15 July 2013 09:48, Steven D'Aprano wrote:

...

On 14/07/13 21:09, Nick Coghlan wrote:

...

Slight adjustment to the proposed wording to ensure completely undocumented modules are also considered private:

-1 on this adjustment. If somebody cannot be bothered writing a

one-line doc

...

...

string:

"This module is private, don't touch."

then they certainly shouldn't be allowed to use up a public name for a private module. I don't think we should be encouraging more private, undocumented modules. (Documentation is valuable even for private modules.)

For context, this arose when I checked PEP 8 after the point was raised on distutils-sig that pip uses a public name for its implementation module (also pip, same as the CLI), but officially exposes no stable public programmatic API.

I don't think we should bless this as official policy, certainly not for the standard library. We can't tell external modules what to do, and possibly there's nothing we can do about existing undocumented private modules, short of renaming them, but I think that at least for new additions to the std lib module names should follow the same single-underscore convention as functions, classes etc. A single leading underscore introduces the private namespace, everything else is public.

I've seen too much undocumented public code to ever assume that lack of documentation implies it is private.

...

At the moment, according to what little PEP 8 has to say about public vs private interfaces, that means a bundled pip would represent a new public API, despite the fact that the only documented interface for pip is the CLI, not the Python module API.

Since we've been burned by people assuming private APIs are public in the past, I figured it made sense to get the actual standards we follow in practice documented properly, *before* anyone further gets the wrong idea from "import pip; help(pip)".

I can tell you that people will do that regardless of what PEP 8 has to say about "not documented == private". Especially as worded. I'm sure I won't be the only person to reason that if a module has a docstring, it is therefore documented.

It takes one character to explicitly document a module as private. That's much better than expecting people to treat their failure to find documentation as an implicit warning.

We're considering bundling the pip CLI for the convenience of educators and beginners, not adding their implementation module to the standard library. That means we need a bright shining line for what constitutes "in the standard library". Yes, it's better if people properly use a leading underscore, but the fact is many don't (including idlelib, as Terry noted) and it becomes a disruptive change to add it later if people don't start with it. The pip and IDLE developers have better things to be doing than changing the names of undocumented APIs just because the application is distributed along with CPython. Cheers, Nick.

-- Steven _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

On Jul 15, 2013, at 09:56 PM, Antoine Pitrou wrote:

If you really want another word (I am personally fine with "private"), "internal" it should be.

I would be fine with "internal", since that's how PEP 8 already classifies such names. :) -Barry