Control adding script path/CWD to sys.path
This is just an idea, which may not in practice be a major problem but can at times be an inconvenience. I thought I had posted this in the "python -m" thread a little while back but checking my history it does't appear I did, I do apologize if this is a duplicate. When running a python script directly, the directory of that script gets added to sys.path. When running as a module "python -m", it looks like an empty string gets added to sys.path, which appears to result in CWD being used. But, depending on the directory structure, modules imported may not be the expected modules. For instance, if developing a package 'mypackage', with some OS specific code organized into an os module or subpackage "mypackage.os", and then running something like "python -m pytest" or "pylint ..." in the directory for the package would cause an "import os" to treat the mypackage/os(.py) module or package as the top level import and cause errors. (I've actually had this happen which is what prompts this idea) There are other solutions that work right now: Don't develop a subpackage or submodule with the same name as a python package or module, or any other package or module that may be on sys.path. I think this is unrealistic as a future addition could result in a package being added that wasn't previously present and there is no way to know the names of all third-party packages Just change the directory and run the script/module from another directory that would not cause an issue. This works great, but seems to go against not presuming where the script is run from or what the CWD is. Install a script in ~/bin or somewhere to do the same at least for modules. This would set the CWD to ~/bin, and not likely to have any conflicting packages or modules, but still presumes that it must not have a conflicting module. What if my ~/bin happens to have other ".py" files marked as executable that is used for something else. It then becomes a top-level module able to be imported, and possibly interfere with other imports. I'd like to propose a python option which would disable the addition of this directory directory. Essentially something to make it possible to say "import the installed module xyz and not perhaps some script or package sitting right beside me also named xyz". Something like "python3 --no-add-cwd-to-path -m pytest". This would allow in cases where needed, to prevent added the script path or CWD to the sys.path list. This could already be done directly in code: import sys ; sys.path.pop(0) But having an option seem to be more flexible as it give the user of the script the ability to say "don't add this current directory to the python path" for existing scripts and modules without having to edit them, create a custom launcher script, or change directories, etc.
Brianvanderburg2 wrote:
It is the intended and the expected behaviour. The working directory is always added to the sys.path. If you have a hierarchy such as a.b.c.d (corresponding to a/b/c/d filesystem tree) and you run the module c from the sub-directory c, you cannot expect that python guesses that c is a sub-package of a and runs the script as if you are calling from folder a. You must call c from directory a and import it with the corresponding a.b.c hierarchy: python -m a.b.c
On 2/24/20, jdveiga@gmail.com <jdveiga@gmail.com> wrote:
It is the intended and the expected behaviour. The working directory is always added to the sys.path.
You mean always in this particular context, i.e. the working directory is added normally when executing a command via -c or a module as a script via -m. When executing a script normally, the script directory gets added, which is reasonably secure. Adding the working directory to sys.path is ok for the interactive shell and -c commands, but I don't understand why it gets added with -m, which is a security hole, and to me an annoyance. It can be disabled with isolated mode, but that's a blunt instrument that disables too much.
Eryk Sun wrote:
Yeah, you are right.
If current directory is not added to the sys.path, how can modules be imported outside PYTHONPATH? https://docs.python.org/3/glossary.html#term-import-path Python cannot guess where your modules are if paths to search for them are not provided. If you run a script from terminal it is assumed that your modules are in the working directory. Is another asumption possible? Maybe you can pass this path via CLI but I think that this is not currently possible. As long as I know, you must include your path in PYTHONPATH or run it from a proper directory. Correct me if I am wrong. On the other hand, can you explain why adding the current directory to importable paths creates a security hole? I am curious. No idea about this.
On Mon, Feb 24, 2020 at 11:51 AM Brianvanderburg2 via Python-ideas < python-ideas@python.org> wrote:
Are you familiar with the -I option for "isolated mode"? https://docs.python.org/3/using/cmdline.html#id2 -I Run Python in isolated mode. This also implies -E and -s. In isolated mode sys.path contains neither the script’s directory nor the user’s site-packages directory. All PYTHON* environment variables are ignored, too. Further restrictions may be imposed to prevent the user from injecting malicious code. New in version 3.4.
Nick Timkovich wrote:
Oooh... Can you provide some examples on the use of -I? I try to use along with -m (`python -I -m a.b`) and get this error: "python: Error while finding module specification for 'a.b' (ModuleNotFoundError: No module named 'a')". On the other hand `python -m a.b -I` just passed -I as a script's CLI argument. Of course, I get the expected results if run `python -I` on terminal.
On 2/24/20, jdveiga@gmail.com <jdveiga@gmail.com> wrote:
This is a use case for -m that requires adding the working directory to sys.path. I work in virtual environments, and I don't navigate into a package and execute modules. The target package is always either in the standard library or installed in site-packages, and the module is executed from the top level. So for me adding the working directory is a feature I never need, and I completely forgot about why anyone would want it.
On Mon, Feb 24, 2020 at 6:04 PM Eryk Sun <eryksun@gmail.com> wrote:
Having the working directory added when using the -m flag is something I use in quite a few of my project. See https://aroberge.github.io/friendly-traceback-docs/docs/html/usage.html for example (example usages 3 to 6 rely on this).
I see... never use this feature. I make extensive use of running modules from local (non-PYTHONPATH) directories. When these modules import (local) sub-modules, the best and simplest method I have found is running them from top directory of the corresponding application. I can run any module in the corresponding hierarchy easily. (Plus running other utilities just providing relative imports.) That is my workflow, so my recommendation on running modules and avoiding import errors.
I guess I'm confused if you want (e.g.) `python json.py` to do something but not `python -m json` to call the same file (a different way), given you're in the directory. nick@lappy386:~/Code$ echo "{}" | python -m json.tool {} nick@lappy386:~/Code$ touch json.py nick@lappy386:~/Code$ echo "{}" | python -m json.tool .../bin/python: Error while finding module specification for 'json.tool' (ModuleNotFoundError: __path__ attribute not found on 'json' while trying to find 'json.tool') nick@lappy386:~/Code$ echo "{}" | python -I -m json.tool {} Isolated mode effectively ignores the environment (variables and directory), so it will only use what's in that Python's site-packages (non-user, but could be a virtualenv) If you want the local path to not be on `sys.path` for a *script-like* call, you could do `python -I myscript.py`, but `-m mypkg` is I think indistinguishable from trying to `import mypkg`, so it needs to be discoverable on `sys.path`...this feels like an A/B problem: what are you trying to do? On Mon, Feb 24, 2020 at 3:35 PM <jdveiga@gmail.com> wrote:
I'll have to check that out. It looks like it would work fine for my purposes since I normally run from virtual envs, but could be restrictive in that the user site-packages isn't added. I also found some other mentions of this: https://bugs.python.org/issue33053https://bugs.python.org/issue13475 One idea may be, when executing "python -m", check to see if the module is loaded from CWD, if so then have CWD added, but if the module is not loaded from CWD then not have it added. -----Original Message----- From: Nick Timkovich <prometheus235@gmail.com> To: Brianvanderburg2 <brianvanderburg2@aim.com> Cc: python-ideas@python.org <python-ideas@python.org> Sent: Mon, Feb 24, 2020 2:52 pm Subject: Re: [Python-ideas] Control adding script path/CWD to sys.path On Mon, Feb 24, 2020 at 11:51 AM Brianvanderburg2 via Python-ideas <python-ideas@python.org> wrote: When running a python script directly, the directory of that script gets added to sys.path. When running as a module "python -m", it looks like an empty string gets added to sys.path, which appears to result in CWD being used. Are you familiar with the -I option for "isolated mode"? https://docs.python.org/3/using/cmdline.html#id2 -I Run Python in isolated mode. This also implies -E and -s. In isolated mode sys.path contains neither the script’s directory nor the user’s site-packages directory. All PYTHON* environment variables are ignored, too. Further restrictions may be imposed to prevent the user from injecting malicious code. New in version 3.4.
Brianvanderburg2 wrote:
It is the intended and the expected behaviour. The working directory is always added to the sys.path. If you have a hierarchy such as a.b.c.d (corresponding to a/b/c/d filesystem tree) and you run the module c from the sub-directory c, you cannot expect that python guesses that c is a sub-package of a and runs the script as if you are calling from folder a. You must call c from directory a and import it with the corresponding a.b.c hierarchy: python -m a.b.c
On 2/24/20, jdveiga@gmail.com <jdveiga@gmail.com> wrote:
It is the intended and the expected behaviour. The working directory is always added to the sys.path.
You mean always in this particular context, i.e. the working directory is added normally when executing a command via -c or a module as a script via -m. When executing a script normally, the script directory gets added, which is reasonably secure. Adding the working directory to sys.path is ok for the interactive shell and -c commands, but I don't understand why it gets added with -m, which is a security hole, and to me an annoyance. It can be disabled with isolated mode, but that's a blunt instrument that disables too much.
Eryk Sun wrote:
Yeah, you are right.
If current directory is not added to the sys.path, how can modules be imported outside PYTHONPATH? https://docs.python.org/3/glossary.html#term-import-path Python cannot guess where your modules are if paths to search for them are not provided. If you run a script from terminal it is assumed that your modules are in the working directory. Is another asumption possible? Maybe you can pass this path via CLI but I think that this is not currently possible. As long as I know, you must include your path in PYTHONPATH or run it from a proper directory. Correct me if I am wrong. On the other hand, can you explain why adding the current directory to importable paths creates a security hole? I am curious. No idea about this.
On Mon, Feb 24, 2020 at 11:51 AM Brianvanderburg2 via Python-ideas < python-ideas@python.org> wrote:
Are you familiar with the -I option for "isolated mode"? https://docs.python.org/3/using/cmdline.html#id2 -I Run Python in isolated mode. This also implies -E and -s. In isolated mode sys.path contains neither the script’s directory nor the user’s site-packages directory. All PYTHON* environment variables are ignored, too. Further restrictions may be imposed to prevent the user from injecting malicious code. New in version 3.4.
Nick Timkovich wrote:
Oooh... Can you provide some examples on the use of -I? I try to use along with -m (`python -I -m a.b`) and get this error: "python: Error while finding module specification for 'a.b' (ModuleNotFoundError: No module named 'a')". On the other hand `python -m a.b -I` just passed -I as a script's CLI argument. Of course, I get the expected results if run `python -I` on terminal.
On 2/24/20, jdveiga@gmail.com <jdveiga@gmail.com> wrote:
This is a use case for -m that requires adding the working directory to sys.path. I work in virtual environments, and I don't navigate into a package and execute modules. The target package is always either in the standard library or installed in site-packages, and the module is executed from the top level. So for me adding the working directory is a feature I never need, and I completely forgot about why anyone would want it.
On Mon, Feb 24, 2020 at 6:04 PM Eryk Sun <eryksun@gmail.com> wrote:
Having the working directory added when using the -m flag is something I use in quite a few of my project. See https://aroberge.github.io/friendly-traceback-docs/docs/html/usage.html for example (example usages 3 to 6 rely on this).
I see... never use this feature. I make extensive use of running modules from local (non-PYTHONPATH) directories. When these modules import (local) sub-modules, the best and simplest method I have found is running them from top directory of the corresponding application. I can run any module in the corresponding hierarchy easily. (Plus running other utilities just providing relative imports.) That is my workflow, so my recommendation on running modules and avoiding import errors.
I guess I'm confused if you want (e.g.) `python json.py` to do something but not `python -m json` to call the same file (a different way), given you're in the directory. nick@lappy386:~/Code$ echo "{}" | python -m json.tool {} nick@lappy386:~/Code$ touch json.py nick@lappy386:~/Code$ echo "{}" | python -m json.tool .../bin/python: Error while finding module specification for 'json.tool' (ModuleNotFoundError: __path__ attribute not found on 'json' while trying to find 'json.tool') nick@lappy386:~/Code$ echo "{}" | python -I -m json.tool {} Isolated mode effectively ignores the environment (variables and directory), so it will only use what's in that Python's site-packages (non-user, but could be a virtualenv) If you want the local path to not be on `sys.path` for a *script-like* call, you could do `python -I myscript.py`, but `-m mypkg` is I think indistinguishable from trying to `import mypkg`, so it needs to be discoverable on `sys.path`...this feels like an A/B problem: what are you trying to do? On Mon, Feb 24, 2020 at 3:35 PM <jdveiga@gmail.com> wrote:
I'll have to check that out. It looks like it would work fine for my purposes since I normally run from virtual envs, but could be restrictive in that the user site-packages isn't added. I also found some other mentions of this: https://bugs.python.org/issue33053https://bugs.python.org/issue13475 One idea may be, when executing "python -m", check to see if the module is loaded from CWD, if so then have CWD added, but if the module is not loaded from CWD then not have it added. -----Original Message----- From: Nick Timkovich <prometheus235@gmail.com> To: Brianvanderburg2 <brianvanderburg2@aim.com> Cc: python-ideas@python.org <python-ideas@python.org> Sent: Mon, Feb 24, 2020 2:52 pm Subject: Re: [Python-ideas] Control adding script path/CWD to sys.path On Mon, Feb 24, 2020 at 11:51 AM Brianvanderburg2 via Python-ideas <python-ideas@python.org> wrote: When running a python script directly, the directory of that script gets added to sys.path. When running as a module "python -m", it looks like an empty string gets added to sys.path, which appears to result in CWD being used. Are you familiar with the -I option for "isolated mode"? https://docs.python.org/3/using/cmdline.html#id2 -I Run Python in isolated mode. This also implies -E and -s. In isolated mode sys.path contains neither the script’s directory nor the user’s site-packages directory. All PYTHON* environment variables are ignored, too. Further restrictions may be imposed to prevent the user from injecting malicious code. New in version 3.4.
participants (5)
-
André Roberge -
Brianvanderburg2 -
Eryk Sun -
jdveiga＠gmail.com -
Nick Timkovich