setuptools: mishandles install of non-python scripts
easy_install doesn't properly handle non-python scripts in a distribution. The problem shows up when using easy_install to install mercurial: <http://cheeseshop.python.org/pypi/Mercurial/> The mercurial distribution includes two scripts, one a python script, the other a shell script: hg - #!/usr/bin/env python hgmerge - #!/bin/sh Using easy_install, both scripts get installed with a python wrapper script. Execution of hgmerge fails when the wrapper tries to run the shell script under python: $ hgmerge Traceback (most recent call last): File "/usr/local/python/bin/hgmerge", line 5, in ? pkg_resources.run_script('mercurial==dbeaa4369121', 'hgmerge') File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-pac kages/setuptools-0.6b3-py2.4.egg/pkg_resources.py", line 407, in run_script self.require(requires)[0].run_script(script_name, ns) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-pac kages/setuptools-0.6b3-py2.4.egg/pkg_resources.py", line 1084, in run_script execfile(script_filename, namespace, namespace) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-pac kages/mercurial-dbeaa4369121-py2.4-macosx-10.4-fat.egg/EGG-INFO/scripts/h gmerge", line 20 if [ -z "$EDITOR" ]; then ^ SyntaxError: invalid syntax If mercurial is installed using distutils directly; python setup.py install both scripts work properly. In build_scripts.py in distutils.command, distuitils uses the results of first_line_re to decide whether to adjust the python value in the shbang line of the script: if no "python" is found, distutils installs the script without modification. easy_install.py uses the first_line_re test from distutils but seems to unconditionally add a wrapper to the script during installation regardless of the script's type. -- Ned Deily, nad@acm.org
At 05:22 PM 6/9/2006 -0700, Ned Deily wrote:
In build_scripts.py in distutils.command, distuitils uses the results of first_line_re to decide whether to adjust the python value in the shbang line of the script: if no "python" is found, distutils installs the script without modification. easy_install.py uses the first_line_re test from distutils but seems to unconditionally add a wrapper to the script during installation regardless of the script's type.
Who needs shell scripts when you have Python? ;) I can (sort of) fix this, but there are some potentially-significant limitations. Notably, if the shell script runs some Python code, it's not guaranteed that the library version it accesses will be the same one that the script was installed with, or that it will be able to access the library at all (e.g., if the user installed it with --multi-version). I'll have to add some kind of warning message when processing non-Python scripts in --multi-version mode.
At 05:22 PM 6/9/2006 -0700, Ned Deily wrote:
easy_install doesn't properly handle non-python scripts in a distribution. ... In build_scripts.py in distutils.command, distuitils uses the results of first_line_re to decide whether to adjust the python value in the shbang line of the script: if no "python" is found, distutils installs the script without modification. easy_install.py uses the first_line_re test from distutils but seems to unconditionally add a wrapper to the script during installation regardless of the script's type.
There's another wrinkle, as it turns out. It's not sufficient to inspect the first line for a #!python presence - it's perfectly valid and possible to have .py or .pyw scripts that don't have a #! line, especially on Windows. So, the actual condition to test if the program is a Python script should be to compile it. If it's valid Python syntax, easy_install will assume it's a Python script, rather than a shell, batch, or other kind of script file. But, it's *also* necessary to check whether there is a *non*-Python #! line, if the first line begins with #!. It's quite possible for a shell script to be syntactically valid Python, especially if you have a shell prolog hidden in what looks like a Python docstring. This is a trick that I believe is listed in the Python FAQ, although I don't know if it's used in any currently-distributed scripts. But it should be supported. So the full rules for analyzing scripts should be: * If there's a #! line, use first_line_re to tell if it's Python * If there's a '.py' or '.pyw' extension, assume it's Python * Otherwise, try to compile the script. If it compiles, assume it's Python * Otherwise, assume it's not. There is one possiblity I'm leaving out here, which is that you have some other script interpreter (e.g. .bat or .cmd files on Windows) that just magically happen to be syntactically-valid Python. In the .bat/.cmd case, anything more complex than a series of one-word commands with no arguments would fail to be valid Python, so it seems incredibly unlikely that this would ever happen. Actually, there's one other possibility I'm also leaving out, which is that you have a .bat or .cmd or some other script whose first line uses -x to skip that line. In such a case, lines 2-N of the file are valid Python, but line 1 doesn't start with #!. In theory, what this should do is keep the first line and wrap the rest. However, such scripts aren't supported by the distutils, so I'm not going to add them to setuptools for 0.6.
participants (2)
-
Ned Deily -
Phillip J. Eby