On Mon, Jan 25, 2021 at 4:18 PM Antoine Pitrou wrote:

On Mon, 25 Jan 2021 14:03:22 +0100 Victor Stinner wrote:

...

The list is opinionated and defined by its documentation:

So "the list is opinionated" means there can be false negatives, i.e. some stdlib modules which are not present in this list?

Test modules of the stdlib are excluded. Example:

...

...

import sys '_testcapi' in sys.module_names # _testcapi extension False 'test' in sys.module_names # Lib/test/ package False import _testcapi _testcapi import test test

It can be changed if it's an issue. That's also why I sent an email to python-dev, to see if there is something wrong with sys.module_names definition. Victor -- Night gathers, and now my watch begins. It shall not end until my death.

On Mon, Jan 25, 2021 at 06:46:51PM +0300, Ivan Pozdeev via Python-Dev wrote:

There's a recurring error case when a 3rd-party module overrides a standard one if it happens to have the same name.

Any argument and expectation is off in this case. We shouldn't worry about such scenarios. -- Senthil

Hello, In general, I love the idea and implementation. I'm not in love with the name though, it makes it sound like it contains all module names imported/available. We have sys.module already containing all module imported. So without a deeper knowledge sys.modules_names is very close to sys.module.keys() or all available modules. Can we name it instead sys.stdlib_modules_names to clarify that this is standard library only subset and not all available modules for the interpreter? Thanks, On Mon, Jan 25, 2021 at 4:33 PM Victor Stinner wrote:

Hi Ivan,

On Mon, Jan 25, 2021 at 4:53 PM Ivan Pozdeev via Python-Dev wrote:

...

Just _names_? There's a recurring error case when a 3rd-party module overrides a standard one if it happens to have the same name. If you filter such a module out, you're shooting yourself in the foot...

Overriding stdlib modules has been discussed in the issue.

For example, it was proposed to add an attribute to all stdlib modules (__stdlib__=True or __author__ = 'PSF'), and then check if the attribute exists or not. The problem is that importing a module to check for its attribute cause side effect or fail, and so cannot be used for some use cases. For example, it would be a surprising to open a web browser window when running isort on a Python code containing "import antigravity". Another problem is that third party can also add the attribute to pretend that their code is part of the stdlib.

In a previous version of my PR, I added a note about sys.path and overriding stdlib modules, but I have been asked to remove it. Feel free to propose a PR to add such note if you consider that it's related to sys.module_names.

Please read the discussion at https://bugs.python.org/issue42955 and https://github.com/python/cpython/pull/24238

Victor _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7HMWTGBE... Code of Conduct: http://python.org/psf/codeofconduct/

On 1/25/21 5:03 AM, Victor Stinner wrote:

I just added a new sys.module_names attribute, list (technically a frozenset) of all stdlib module names

The list is opinionated and defined by its documentation

For packages, only sub-packages are listed, not sub-modules. For example, ``concurrent`` package and ``concurrent.futures`` sub-package are listed, but not ``concurrent.futures.base`` sub-module.

I'm not sure I understand the above. Is it fair to say that any stdlib module, except for private or test (./Lib/test/*) modules, that can be imported are listed in `sys.module_names`? My confusion stems from being able to import `concurrent.futures` but not `concurrent.futures.base`. -- ~Ethan~

On Mon, Jan 25, 2021 at 6:39 PM Ethan Furman wrote:

...

For packages, only sub-packages are listed, not sub-modules. For example, ``concurrent`` package and ``concurrent.futures`` sub-package are listed, but not ``concurrent.futures.base`` sub-module.

I'm not sure I understand the above. Is it fair to say that any stdlib module, except for private or test (./Lib/test/*) modules,

Private modules are listed: __future__, _abc, _aix_support, etc.

that can be imported are listed in `sys.module_names`? My confusion stems from being able to import `concurrent.futures` but not `concurrent.futures.base`.

For package, I chose to exclude sub-modules just to keep the list short. ~300 items can be displayed and read manually. If you want to check if "asyncio.base_events" is a stdlib module, extract "asyncio" string and check if "asyncio" is part of the list. sys.module_names cannot be used directly if you need to get the exhaustive list of all modules including sub-modules. pkgutil.iter_modules() can be used to list modules of package:

...

...

[mod.name for mod in pkgutil.iter_modules(path=asyncio.__path__)] ['__main__', 'base_events', 'base_futures', 'base_subprocess', 'base_tasks', 'constants', 'coroutines', 'events', 'exceptions', 'format_helpers', 'futures', 'locks', 'log', 'mixins', 'proactor_events', 'protocols', 'queues', 'runners', 'selector_events', 'sslproto', 'staggered', 'streams', 'subprocess', 'tasks', 'threads', 'transports', 'trsock', 'unix_events', 'windows_events', 'windows_utils']

One drawback is that if the stdlib would contain packages without __init__.py file, a third party project could add a sub-module to it (ex: inject encodings/myencoding.py in the encodings package). But it seems like all Lib/ sub-directories contain an __init__.py file, so it's not an issue in practice. If we include sub-modules, sys.module_names grows from 312 names to 813 names (2.6x more). Two examples: "collections", +"collections.abc", "concurrent", "concurrent.futures", +"concurrent.futures._base", +"concurrent.futures.process", +"concurrent.futures.thread", Just the encodings package contains 121 sub-modules. Victor -- Night gathers, and now my watch begins. It shall not end until my death.

That's not possible. Stdlib can be arranged any way a user/maintainer wishes (zipped stdlib and virtual environments are just two examples), so there's no way to tell if the module's location is "right". Dowstream changes are also standard practice so there's no way to verify a module's contents, either. As such, there's no way to tell if any given module being imported is a standard or a 3rd-party one. On 25.01.2021 20:33, Chris Jerdonek wrote:

On Mon, Jan 25, 2021 at 7:51 AM Ivan Pozdeev via Python-Dev mailto:python-dev@python.org> wrote:

Just _names_? There's a recurring error case when a 3rd-party module overrides a standard one if it happens to have the same name. If you filter such a module out, you're shooting yourself in the foot...

Would another use case be to support issuing a warning if a third-party module is imported whose name matches a standard one? A related use case would be to build on this and define a function that accepts an already imported module and return whether it is from the standard library. Unlike, the module_names attribute, this function would reflect the reality of the underlying module, and so not have false positives as with doing a name check alone.

—Chris

On 25.01.2021 16:03, Victor Stinner wrote: > Hi, > > I just added a new sys.module_names attribute, list (technically a > frozenset) of all stdlib module names: > https://bugs.python.org/issue42955 https://bugs.python.org/issue42955 > > There are multiple use cases: > > * Group stdlib imports when reformatting a Python file, > * Exclude stdlib imports when computing dependencies. > * Exclude stdlib modules when listing extension modules on crash or > fatal error, only list 3rd party extension (already implemented in > master, see bpo-42923 ;-)). > * Exclude stdlib modules when tracing the execution of a program using > the trace module. > * Detect typo and suggest a fix: ImportError("No module named maths. > Did you mean 'math'?",) (test the nice friendly-traceback project!). > > Example: > >>>> 'asyncio' in sys.module_names > True >>>> 'numpy' in sys.module_names > False > >>>> len(sys.module_names) > 312 >>>> type(sys.module_names) > > >>>> sorted(sys.module_names)[:10] > ['__future__', '_abc', '_aix_support', '_ast', '_asyncio', '_bisect', > '_blake2', '_bootsubprocess', '_bz2', '_codecs'] >>>> sorted(sys.module_names)[-10:] > ['xml.dom', 'xml.etree', 'xml.parsers', 'xml.sax', 'xmlrpc', 'zipapp', > 'zipfile', 'zipimport', 'zlib', 'zoneinfo'] > > The list is opinionated and defined by its documentation: > >     A frozenset of strings containing the names of standard library >     modules. > >     It is the same on all platforms. Modules which are not available on >     some platforms and modules disabled at Python build are also listed. >     All module kinds are listed: pure Python, built-in, frozen and >     extension modules. Test modules are excluded. > >     For packages, only sub-packages are listed, not sub-modules. For >     example, ``concurrent`` package and ``concurrent.futures`` >     sub-package are listed, but not ``concurrent.futures.base`` >     sub-module. > >     See also the :attr:`sys.builtin_module_names` list. > > The design (especially, the fact of having the same list on all > platforms) comes from the use cases list above. For example, running > isort should produce the same output on any platform, and not depend > if the Python stdlib was splitted into multiple packages on Linux > (which is done by most popular Linux distributions). > > The list is generated by the Tools/scripts/generate_module_names.py script: > https://github.com/python/cpython/blob/master/Tools/scripts/generate_module_... https://github.com/python/cpython/blob/master/Tools/scripts/generate_module_... > > When you add a new module, you must run "make regen-module-names, > otherwise a pre-commit check will fail on your PR ;-) The list of > Windows extensions is currently hardcoded in the script (contributions > are welcomed to discover them, since the list is short and evolves > rarely, I didn't feel the need to spend time that on that). > > Currently (Python 3.10.0a4+), there are 312 names in sys.module_names, > stored in Python/module_names.h: > https://github.com/python/cpython/blob/master/Python/module_names.h https://github.com/python/cpython/blob/master/Python/module_names.h > > It was decided to include "helper" modules like "_aix_support" which > is used by sysconfig. But test modules like _testcapi are excluded to > make the list shorter (it's rare to run the CPython test suite outside > Python). > > There are 83 private modules, name starting with an underscore > (exclude _abc but also __future__): > >>>> len([name for name in sys.module_names if not name.startswith('_')]) > 229 > > This new attribute may help to define "what is the Python stdlib" ;-) > > Victor

-- Regards, Ivan _______________________________________________ Python-Dev mailing list -- python-dev@python.org mailto:python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org mailto:python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KCJDHKOK... https://mail.python.org/archives/list/python-dev@python.org/message/KCJDHKOK... Code of Conduct: http://python.org/psf/codeofconduct/ http://python.org/psf/codeofconduct/

-- Regards, Ivan

Fortunately for, you :) , all this argument is not against the feature per se but only against its use to blindly filter module lists for automated bug reports. On 26.01.2021 1:34, Victor Stinner wrote:

On Mon, Jan 25, 2021 at 11:22 PM Ivan Pozdeev via Python-Dev wrote:

...

That's not possible.

Stdlib can be arranged any way a user/maintainer wishes (zipped stdlib and virtual environments are just two examples), so there's no way to tell if the module's location is "right". Dowstream changes are also standard practice so there's no way to verify a module's contents, either.

As such, there's no way to tell if any given module being imported is a standard or a 3rd-party one. By the way, IMO it's also a legit use case on an old Python version to override a stdlib module with a patched or more recent version, to get a bugfix for example ;-) Even if it's an uncommon use case, it can solve some practical issues.

Victor

-- Regards, Ivan

On Mon, Jan 25, 2021 at 2:05 PM Victor Stinner wrote:

...

On Mon, Jan 25, 2021 at 7:51 AM Ivan Pozdeev via Python-Dev <

...

...

Just _names_? There's a recurring error case when a 3rd-party module

overrides a standard one if it happens to have the same name. If you

...

filter such a module out, you're shooting yourself in the foot...

Would another use case be to support issuing a warning if a third-party module is imported whose name matches a standard one? A related use case would be to build on this and define a function that accepts an already imported module and return whether it is from the standard library. Unlike,

On Mon, Jan 25, 2021 at 6:37 PM Chris Jerdonek wrote: python-dev@python.org> wrote: the module_names attribute, this function would reflect the reality of the underlying module, and so not have false positives as with doing a name check alone.

This is a different use case which requires a different solution. sys.module_names solve some specific use cases (that I listed in my first email).

In Python 3.9, you can already check if a module __file__ is in the sysconfig.get_paths()['stdlib'] directory. You don't need to modify Python for that.

But to issue a warning when a standard module is being overridden like I was suggesting, wouldn’t you also need to know whether the name of the module being imported is a standard name, which is what says.module_names provides? —Chris If you also would like to check if an *extension* module comes from

the stdlib, you need to get the "lib-dynload" directory. I failed to find a programmatic way to get this directory, maybe new API would be needed for that.

Victor -- Night gathers, and now my watch begins. It shall not end until my death.

On Tue, 26 Jan 2021 10:36:10 +1100 Steven D'Aprano wrote:

On Mon, Jan 25, 2021 at 06:17:09PM +0100, Victor Stinner wrote:

...

Hi Bernat,

"stdlib_module_names" was my first idea but it looks too long, so I chose "module_names". But someone on Twitter and now you asked me why not "stdlib_module_names", so I wrote a PR to rename module_names to sys.stdlib_module_names: https://github.com/python/cpython/pull/24332

At least "stdlib_module_names" better summarizes its definition: "A frozenset of strings containing the names of standard library modules".

Your first instinct that it is too long is correct.

Disagreed. This is niche enough that it warrants a long but explicit name, rather than some ambiguous shortcut.

Just call it "stdlib" or "stdlib_names".

If you call it "stdlib", then you should make it a namedtuple that will expose various information, such as "sys.stdlib.module_names". Regards Antoine.

On Tue, Jan 26, 2021 at 12:44 AM Steven D'Aprano wrote:

On Mon, Jan 25, 2021 at 06:17:09PM +0100, Victor Stinner wrote:

...

Hi Bernat,

"stdlib_module_names" was my first idea but it looks too long, so I chose "module_names". But someone on Twitter and now you asked me why not "stdlib_module_names", so I wrote a PR to rename module_names to sys.stdlib_module_names: https://github.com/python/cpython/pull/24332

At least "stdlib_module_names" better summarizes its definition: "A frozenset of strings containing the names of standard library modules".

Your first instinct that it is too long is correct. Just call it "stdlib" or "stdlib_names". The fact that it is a frozen set of module names will be obvious from just looking at it, and there is no need for the name to explain everything about it. We have:

The sys module already has a sys.modules attribute, and so sys.module_names sounds like "give me the name of all imported modules": sys.module.keys(). It's confusing. Just after I announced the creation of the attribute, at least 3 people told me that they were confused by the name. Also, my PR was approved quickly 3 times which confirms that the rename was a good idea ;-) In general, I agree that short names are great ;-) For example, I like short obj.name() rather than obj.getname() when it doesn't make sense to set the name. Naming is a hard problem :-D Victor -- Night gathers, and now my watch begins. It shall not end until my death.

If the length of the name is any kind of issue, since the stdlib only contains modules (and packages), why not just sys.stdlib_names? On Mon, Jan 25, 2021 at 5:18 PM Victor Stinner wrote:

Hi Bernat,

"stdlib_module_names" was my first idea but it looks too long, so I chose "module_names". But someone on Twitter and now you asked me why not "stdlib_module_names", so I wrote a PR to rename module_names to sys.stdlib_module_names: https://github.com/python/cpython/pull/24332

At least "stdlib_module_names" better summarizes its definition: "A frozenset of strings containing the names of standard library modules".

Victor

On Mon, Jan 25, 2021 at 5:39 PM Bernat Gabor wrote:

...

Hello,

In general, I love the idea and implementation. I'm not in love with the

name though, it makes it sound like it contains all module names imported/available. We have sys.module already containing all module imported. So without a deeper knowledge sys.modules_names is very close to sys.module.keys() or all available modules. Can we name it instead sys.stdlib_modules_names to clarify that this is standard library only subset and not all available modules for the interpreter?

...

Thanks,

On Mon, Jan 25, 2021 at 4:33 PM Victor Stinner

wrote:

...

...

Hi Ivan,

On Mon, Jan 25, 2021 at 4:53 PM Ivan Pozdeev via Python-Dev wrote:

...

Just _names_? There's a recurring error case when a 3rd-party module

overrides a standard one if it happens to have the same name. If you

...

...

filter such a module out, you're shooting yourself in the foot...

Overriding stdlib modules has been discussed in the issue.

For example, it was proposed to add an attribute to all stdlib modules (__stdlib__=True or __author__ = 'PSF'), and then check if the attribute exists or not. The problem is that importing a module to check for its attribute cause side effect or fail, and so cannot be used for some use cases. For example, it would be a surprising to open a web browser window when running isort on a Python code containing "import antigravity". Another problem is that third party can also add the attribute to pretend that their code is part of the stdlib.

In a previous version of my PR, I added a note about sys.path and overriding stdlib modules, but I have been asked to remove it. Feel free to propose a PR to add such note if you consider that it's related to sys.module_names.

Please read the discussion at https://bugs.python.org/issue42955 and https://github.com/python/cpython/pull/24238

Victor _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7HMWTGBE... Code of Conduct: http://python.org/psf/codeofconduct/

-- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/WJMYK2JK... Code of Conduct: http://python.org/psf/codeofconduct/

On Tue, Jan 26, 2021 at 10:04 PM Steve Dower wrote:

On 1/26/2021 8:32 PM, Steve Holden wrote:

...

If the length of the name is any kind of issue, since the stdlib only contains modules (and packages), why not just sys.stdlib_names?

And since the modules can vary between platforms and builds, why wouldn't this be sysconfig.stdlib_names rather than sys.stdlib_names?

The list is the same on all platforms on purpose ;-) Example:

...

...

'winsound' in sys.stdlib_module_names True 'ossaudiodev' in sys.stdlib_module_names True

For example, grouping stdlib imports using sys.stdlib_module_names gives the same output on any platform, even if there were missing dependencies when you built Python. Victor -- Night gathers, and now my watch begins. It shall not end until my death.

On Tue, Jan 26, 2021 at 12:08:03PM -0800, Brett Cannon wrote:

On Tue, Jan 26, 2021 at 1:26 AM Antoine Pitrou wrote:

[...]

...

Disagreed. This is niche enough that it warrants a long but explicit name, rather than some ambiguous shortcut.

I agree w/ Antoine that a more descriptive name for such a niche (but useful!) attribute makes sense.

This descriptive name is *literally incorrect*. By design, it doesn't list modules. It only lists sub-packages and not sub-modules, to keep the number of entries more managable. (Personally, I don't think an extra hundred or two names makes that much difference. Its going to be a big list one way or the other.) So by the current rules, many stdlib modules are not included and the name is inaccurate. If you're not going to list all the dotted modules of a package, why distinguish sub-modules from sub-packages? It is confusing and ackward to have only some dotted modules listed based on the **implementation**. (We need a good term for "things you can import that create a module *object* regardless of whether they are a *module file* or a *package*. I'm calling them a dotted module for lack of a better name.) By the current rules, many stdlib modules are not listed, and you can't see why unless you know their implementation: * urllib - listed * urllib.parse - not listed * collections - listed * collections.abc - not listed * email - listed * email.parser - not listed * email.mime - listed # Surprise! So we have this weird situation where an implementation detail of the dotted module (whether it is a file `package/module.py` or `package/module/__init__.py`) determines whether it shows up or not. And because the file system structure of a module is not part of its API, that implementation detail could change without warning. I think that either of: 1. list *all* package dotted modules regardless of whether they are implemented as a sub-module or sub-package; 2. list *no* package dotted modules, only the top-level package; would be better than this inconsistent hybrid of only listing some dotted modules. (Excluding the test modules is fine.) -- Steve

On Tue, Jan 26, 2021 at 10:56:57AM +0100, Victor Stinner wrote:

On Tue, Jan 26, 2021 at 12:44 AM Steven D'Aprano wrote:

[...]

...

Your first instinct that it is too long is correct. Just call it "stdlib" or "stdlib_names". The fact that it is a frozen set of module names will be obvious from just looking at it, and there is no need for the name to explain everything about it. We have:

The sys module already has a sys.modules attribute, and so sys.module_names sounds like "give me the name of all imported modules": sys.module.keys().

Then its a good thing I didn't propose calling it "module_names" :-) -- Steve

On Wed, Jan 27, 2021 at 10:44:00AM +0100, Antoine Pitrou wrote:

Ok, then "stdlib_package_names"? :-)

Heh :-) I see your smiley, and I'm not going to argue about the name any further. I have my preference, but if the consensus is stdlib_module_names, so be it. But I think the inconsistency between sub-modules and sub-packages is important. We should either list all sub-whatever or none of them, rather than only some of them. -- Steve

On Wed, Jan 27, 2021 at 1:16 AM Steven D'Aprano wrote:

Right. This is (I think) Steve's point: the list is inaccurate, because the existence of 'winsound' in the stdlib_module_names doesn't mean that the module 'winsound' exists.

This point is addressed by the definition of the list: sys.stdlib_module_names documentation. "It is the same on all platforms. Modules which are not available on some platforms and modules disabled at Python build are also listed. All module kinds are listed: pure Python, built-in, frozen and extension modules. Test modules are excluded." https://docs.python.org/dev/library/sys.html#sys.stdlib_module_names As I wrote previously, there are use cases which *require* the list being the same on all platforms. Moreover, in practice, it's quite hard to build a list of available stdlib module names. You need to build extension modules, try to implement them, then rebuild the list of module which requires to rebuild Python. It's not convenient. Also, there are different definition of "available". For example, "import multiprocessing" can fail on some platforms if there is no lock implementation available. It's not because it's installed on the system that the import will work for sure. IMO the only reliable way to check if a module can be imported... is to import it. And then you hit again the issue of import side effects. There are different ways to filter sys.stdlib_module_names list to only list "available" modules. Try import, pkgutil.iter_modules() or pkgutil.walk_packages(). IMO it should remain out of the scope of sys.stdlib_module_names. Victor -- Night gathers, and now my watch begins. It shall not end until my death.

I see a bunch of similar -- but not quite the same -- use cases. I feel like instead of a set, it should be a dict pointing to an object with attributes that describe the module in various ways (top-level vs subpackage, installed on this machine or not, test module or not, etc). I'll understand if this seems like scope creep, but try not to rule it out as a future enhancement. (e.g., don't promise it will be precisely a set., as opposed to the keys of a map.)