Generalize sys.executable replacement from scripts to data_files (or more)
This was opened as an issue (https://github.com/pypa/pip/issues/4032) some years ago, but it was more recently recommended to open a thread here and that issue was closed due to inactivity, so here goes. pip currently rewrites sys.executable in the shebang line of scripts to be installed from wheels. This is necessary for portability, envs, etc. On the far opposite end of the spectrum, conda finds and rewrites the install prefix in any file anywhere in the package (including binaries). In IPython/Jupyter (specifically ipykernel), we have a file that we want to install that is a data_file in `share/jupyter/kernels/python3/kernel.json`. Like scripts, this file contains a reference to sys.executable, and like scripts, this should be rewritten at install time, since the wheel doesn't know the right value at build time. However, the sys.executable rewrite from pip is not configurable or accessible, so only scripts are allowed to have this modification. It would be useful to us to be able to opt-in to this modification for data_files as well. Our current choices are: 1. disable wheels for our package because we need to know sys.executable to create these files, and thus must rely on arbitrary code execution to get it right 2. use `python` and add special runtime-handling (this is what we do, and it can be wrong, e.g. when sys.executable at runtime is not actually the sys.executable used for installation, exactly the problem script shebang rewrite solves for scripts) 3. don't install the kernelspec with the package and tell users that proper installation is a two step process: `pip install ipykernel; ipython kernel install --sys-prefix`. We used to do this, and it caused lots of problems due to uninstall/upgrade causing the package and kernelspec to fall out of sync since part of the package is not managed by the package manager. A "works for us" version of this could be to allow specifying that a given data_file is a script (or entrypoint) and should be treated as one. This would allow us to place a script next to our json file in a way that it is found by our kernelspec mechanism. This isn't as general a solution, but it works for us without needing to generalize sys.executable (or sys.prefix) handling as much to locations in files other than the shebang, it only needs generalizing the installation location of scripts.
On Fri, 8 Feb 2019 at 20:46, Min RK <benjaminrk@gmail.com> wrote:
This was opened as an issue (https://github.com/pypa/pip/issues/4032) some years ago, but it was more recently recommended to open a thread here and that issue was closed due to inactivity, so here goes.
pip currently rewrites sys.executable in the shebang line of scripts to be installed from wheels. This is necessary for portability, envs, etc. On the far opposite end of the spectrum, conda finds and rewrites the install prefix in any file anywhere in the package (including binaries). In IPython/Jupyter (specifically ipykernel), we have a file that we want to install that is a data_file in `share/jupyter/kernels/python3/kernel.json`. Like scripts, this file contains a reference to sys.executable, and like scripts, this should be rewritten at install time, since the wheel doesn't know the right value at build time. However, the sys.executable rewrite from pip is not configurable or accessible, so only scripts are allowed to have this modification. It would be useful to us to be able to opt-in to this modification for data_files as well.
Our current choices are:
1. disable wheels for our package because we need to know sys.executable to create these files, and thus must rely on arbitrary code execution to get it right 2. use `python` and add special runtime-handling (this is what we do, and it can be wrong, e.g. when sys.executable at runtime is not actually the sys.executable used for installation, exactly the problem script shebang rewrite solves for scripts) 3. don't install the kernelspec with the package and tell users that proper installation is a two step process: `pip install ipykernel; ipython kernel install --sys-prefix`. We used to do this, and it caused lots of problems due to uninstall/upgrade causing the package and kernelspec to fall out of sync since part of the package is not managed by the package manager.
A "works for us" version of this could be to allow specifying that a given data_file is a script (or entrypoint) and should be treated as one. This would allow us to place a script next to our json file in a way that it is found by our kernelspec mechanism. This isn't as general a solution, but it works for us without needing to generalize sys.executable (or sys.prefix) handling as much to locations in files other than the shebang, it only needs generalizing the installation location of scripts.
Unfortunately, the challenge you would face with either of those approaches is that there would then be a large set of package installer versions that will *appear* to install ipykernel correctly, but will in fact miss rewriting the sys.executable reference in the relevant file. If the new semantics are instead specified in a way that forces old installers to fail to install them, that's problematic for a completely different reason: you have to choose between allowing users to continue using old installers and using the new mechanism that you want to use. While it's a pretty gross hack, would it be feasible for ipykernel to change the "python" marker logic to look up its own RECORD file to find a rewritten script file and extract the shebang line from that file? Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (2)
-
Min RK -
Nick Coghlan