Setuptools Bug: all files installed +x
On both linux & OS X, Setuptools installs all .py/.pyc files with mode a+x (executable for all users). This occurs regardless of original the permissions in the source tarball. Doing so breaks nosetests, which by default refuses to import executable files for test-discovery purposes as a safety measure. This behavior is broken & dangerous. -- Pete pfein@pobox.com
At 03:24 PM 4/21/2008 -0400, Pete wrote:
On both linux & OS X, Setuptools installs all .py/.pyc files with mode a+x (executable for all users). This occurs regardless of original the permissions in the source tarball. Doing so breaks nosetests, which by default refuses to import executable files for test-discovery purposes as a safety measure.
This behavior is broken & dangerous.
I don't see how it's either one. An explanation would be helpful. Note, by the way, that setuptools is not particularly designed to support running tests against an installed package; I myself have stopped distributing tests in installed packages and require a source installation (e.g. using easy_install --editable) to run tests. This pattern is particularly important for reducing runtime installation requirements, as tests often require additional packages (such as nose itself) in order to run.
On Apr 21, 2008, at 3:42 PM, Phillip J. Eby wrote:
At 03:24 PM 4/21/2008 -0400, Pete wrote:
On both linux & OS X, Setuptools installs all .py/.pyc files with mode a+x (executable for all users). This occurs regardless of original the permissions in the source tarball. Doing so breaks nosetests, which by default refuses to import executable files for test- discovery purposes as a safety measure.
This behavior is broken & dangerous.
I don't see how it's either one. An explanation would be helpful.
It's broken in that the source tarball includes per-file permissions and setuptools is blindly overriding them. I realize that's simply restating my original complaint, but seeing as setuptools must be *explicitly* changing the permissions on the installed files, perhaps the onus is on you to explain why that's a good idea in the first place. In any event, a motivating example: Some non-script modules are intended to be executable - think doctest, or anything else that does a `if __name__ == __main__:`. As a developer, I purposely set such modules executable (including setting svn:executable) and leave the others as r-w. And there lies the danger. The executable bit is an indication that a file is intended to be executable. Unix-like systems will treat running a file without a leading #! as a shell script. This can cause arbitrary commands to be executed - for example, this is valid python: rm -f /usr Perhaps contrived, but should demonstrate the point. As a more realistic example, `import` is an imagelib command that takes over the X cursor (for taking a screenshot IIRC).
Note, by the way, that setuptools is not particularly designed to support running tests against an installed package; I myself have stopped distributing tests in installed packages and require a source installation (e.g. using easy_install --editable) to run tests.
I'm not looking for explicit testing support from setuptools for testing here - I'm just asking that a bug that breaks a 3rd party testing package be fixed. -- Pete pfein@pobox.com
At 04:23 PM 4/21/2008 -0400, Pete wrote:
I'm not looking for explicit testing support from setuptools for testing here - I'm just asking that a bug that breaks a 3rd party testing package be fixed.
You haven't stated anything yet that sounds like an actual bug to me. What nose is doing seems like the bug to me, in fact, since modules exist "in the wild" that are both module and script. (You yourself gave this as an example.) It certainly might be reasonable... in a posix-only + svn-only + tar-only world... to make the assumption that installed permissions could be based in part on source file permissions. However, setuptools has to support scenarios where none of those environmental conditions apply. (e.g. Windows, a half-dozen revision control systems, and zip files.) That means that existing executable permissions are not -- and as far as I can see, *cannot* -- be used as a cross-platform basis for determining the permissions to give to installed files. Some other mechanism to explicitly specify the permissions would need to exist.
"Phillip J. Eby" <pje@telecommunity.com> writes:
At 03:24 PM 4/21/2008 -0400, Pete wrote:
On both linux & OS X, Setuptools installs all .py/.pyc files with mode a+x (executable for all users). This occurs regardless of original the permissions in the source tarball. Doing so breaks nosetests, which by default refuses to import executable files for test-discovery purposes as a safety measure.
This behavior is broken & dangerous.
I don't see how it's either one. An explanation would be helpful.
Broken: The source tarball has specific permissions set for those files, and Setupttols is breaking that by overriding the permissions. A feature (preserve desired permission mode of the file) is lost. Dangerous: The tests will run fine on the developer's machine, but will skip over the modules entirely when they are installed via Setuptools, because of the above broken behaviour. Test output will indicate no problems, because the permission modes (as incorrectly set by Setuptools) indicate explicitly to the testing tool that those modules should be skipped. This results in tests being omitted silently, which is the danger. (good sigmonster, have a cookie) -- \ “Ignorance more frequently begets confidence than does | `\ knowledge.” —Charles Darwin, _The Descent of Man_, 1871 | _o__) | Ben Finney
On Apr 21, 2008, at 6:01 PM, Phillip J. Eby wrote:
At 04:23 PM 4/21/2008 -0400, Pete wrote:
I'm not looking for explicit testing support from setuptools for testing here - I'm just asking that a bug that breaks a 3rd party testing package be fixed.
You haven't stated anything yet that sounds like an actual bug to me.
What about the dangerous & broken complaint?
What nose is doing seems like the bug to me, in fact, since modules exist "in the wild" that are both module and script. (You yourself gave this as an example.)
No, it's a safety mechanism. From the nose docs for the --exe flag (which forcibly disables such checks): --exe Look for tests in python modules that are executable. Normal behavior is to exclude executable modules, since they may not be import-safe [NOSE_INCLUDE_EXE]
It certainly might be reasonable... in a posix-only + svn-only + tar-only world... to make the assumption that installed permissions could be based in part on source file permissions. However, setuptools has to support scenarios where none of those environmental conditions apply. (e.g. Windows, a half-dozen revision control systems, and zip files.)
I fail to see how setting +x on all non-script .py's addresses the environments that don't support permissions. Given the complete lack of permissions information in non-tar archives, it'd arguably be equally valid to set -x (non-executable) on all .py's or to set them at random. What problem does the current behavior solve? If authors need support for permissions, they clearly need to use an archive file format that supports that. If they don't use such a format, they should expect what they get. For those of us using tar/ posix, the fact that setuptools goes out its way to override developer/ distributor intent is problematic. I know what I'm doing, and setuptools doesn't let me do it.
That means that existing executable permissions are not -- and as far as I can see, *cannot* -- be used as a cross-platform basis for determining the permissions to give to installed files. Some other mechanism to explicitly specify the permissions would need to exist.
If there's a strong reason for +x'ing non-tar archives, perhaps setuptools could disable this behavior for archive formats that support (ie, tar). -- Pete pfein@pobox.com
At 11:49 AM 4/22/2008 -0400, Pete wrote:
On Apr 21, 2008, at 6:01 PM, Phillip J. Eby wrote:
At 04:23 PM 4/21/2008 -0400, Pete wrote:
I'm not looking for explicit testing support from setuptools for testing here - I'm just asking that a bug that breaks a 3rd party testing package be fixed.
You haven't stated anything yet that sounds like an actual bug to me.
What about the dangerous & broken complaint?
Which I don't yet understand, let alone agree with. Simply asserting over and over that it's bad and dangerous doesn't help. One thing that you particularly seem to be missing is that the distutils also ignore a Python module's source permissions -- whether they come from a tarball or not.
On Apr 22, 2008, at 12:19 PM, Phillip J. Eby wrote:
At 11:49 AM 4/22/2008 -0400, Pete wrote:
On Apr 21, 2008, at 6:01 PM, Phillip J. Eby wrote:
At 04:23 PM 4/21/2008 -0400, Pete wrote:
I'm not looking for explicit testing support from setuptools for testing here - I'm just asking that a bug that breaks a 3rd party testing package be fixed.
You haven't stated anything yet that sounds like an actual bug to me.
What about the dangerous & broken complaint?
Which I don't yet understand, let alone agree with. Simply asserting over and over that it's bad and dangerous doesn't help.
This bit, from my email on April 21, 2008 4:23:09; Ben Finney's point about tests being silently skipped is also valid, and was how I originally came across this problem. In any event, a motivating example: Some non-script modules are intended to be executable - think doctest, or anything else that does a `if __name__ == __main__:`. As a developer, I purposely set such modules executable (including setting svn:executable) and leave the others as r-w. And there lies the danger. The executable bit is an indication that a file is intended to be executable. Unix-like systems will treat running a file without a leading #! as a shell script. This can cause arbitrary commands to be executed - for example, this is valid python: rm -f /usr Perhaps contrived, but should demonstrate the point. As a more realistic example, `import` is an imagelib command that takes over the X cursor (for taking a screenshot IIRC).
One thing that you particularly seem to be missing is that the distutils also ignore a Python module's source permissions -- whether they come from a tarball or not.
Ok, but AFAIK distutils doesn't then +x everything, which is the problem here. -- Pete pfein@pobox.com
Am Montag, den 21.04.2008, 18:01 -0400 schrieb Phillip J. Eby:
It certainly might be reasonable... in a posix-only + svn-only + tar-only world... to make the assumption that installed permissions could be based in part on source file permissions. However, setuptools has to support scenarios where none of those environmental conditions apply. (e.g. Windows, a half-dozen revision control systems, and zip files.)
Neither Windows nor zip files support permissions at all, correct? Thus neither one is a reason to mess with the permissions, right? As for the revision control systems: I'd like to learn about those that don't support permissions, because I am not aware of any and I'd like to avoid them in the future. Thanks for your help in this regard, Phillip! Klaus
On Tue, 2008-04-22 at 12:19 -0400, Phillip J. Eby wrote:
At 11:49 AM 4/22/2008 -0400, Pete wrote:
On Apr 21, 2008, at 6:01 PM, Phillip J. Eby wrote:
At 04:23 PM 4/21/2008 -0400, Pete wrote:
I'm not looking for explicit testing support from setuptools for testing here - I'm just asking that a bug that breaks a 3rd party testing package be fixed.
You haven't stated anything yet that sounds like an actual bug to me.
What about the dangerous & broken complaint?
Which I don't yet understand, let alone agree with. Simply asserting over and over that it's bad and dangerous doesn't help.
I thought his explanation was rather clear. Here's a valid Python file that will make you unhappy if you execute it from the shell: rm = 0 # shell error, no effect rf = 1 # another shell error home = 8 # etc... result=`rm -rf /home` In Python you get '0', in Bash you get a wiped out home directory. Yes, it's quite contrived. But his point about "import" also being a typical command on *nix (runs an Imagemagick binary) points out that there's probably even more concerns that we aren't thinking of. This is especially a concern because shell scripts don't abort on error (so you could conceivably make it quite a ways through a Python file until you hit some random name that happens to be a valid Unix command). It's probably unlikely that someone would accidentally run one of these modules inadvertently. But "highly unlikely" is also not "secure". I've certainly run the wrong thing on accident because Bash's tab-completion caught me off-guard. He's also right in that arbitrarily setting the execute bit apparently serves no explainable purpose (otherwise I assume you'd have provided an explanation by now), so the onus of explaining why this is a desirable behavior comes back to setuptools.
One thing that you particularly seem to be missing is that the distutils also ignore a Python module's source permissions -- whether they come from a tarball or not.
I'm not sure why any file in site-packages would need the execute bit set. Executable scripts are (or should be) installed elsewhere. If the bit *must* be set one way or the other, then it should be turned off by default. Regards, Cliff
On Apr 22, 2008, at 3:56 PM, Cliff Wells wrote:
On Tue, 2008-04-22 at 12:19 -0400, Phillip J. Eby wrote:
At 11:49 AM 4/22/2008 -0400, Pete wrote:
On Apr 21, 2008, at 6:01 PM, Phillip J. Eby wrote:
At 04:23 PM 4/21/2008 -0400, Pete wrote:
I'm not looking for explicit testing support from setuptools for testing here - I'm just asking that a bug that breaks a 3rd party testing package be fixed.
You haven't stated anything yet that sounds like an actual bug to me.
What about the dangerous & broken complaint?
Which I don't yet understand, let alone agree with. Simply asserting over and over that it's bad and dangerous doesn't help.
He's also right in that arbitrarily setting the execute bit apparently serves no explainable purpose (otherwise I assume you'd have provided an explanation by now), so the onus of explaining why this is a desirable behavior comes back to setuptools.
Did a decision ever get made here? I don't mean to be a nag, but this is continuing to cause problems for me and my users. Just wanted to make sure it doesn't get lost/forgotten. This is why people have bug trackers y'know. ;-(
On May 1, 2008, at 12:00 PM, Pete wrote:
Did a decision ever get made here? I don't mean to be a nag, but this is continuing to cause problems for me and my users. Just wanted to make sure it doesn't get lost/forgotten.
This is why people have bug trackers y'know. ;-(
I agree that bug trackers are important for this sort of reason. You're welcome to use the one that I set up: http://allmydata.org/trac/setuptools/report/1 PJE hasn't agreed to use it so far, but on the other hand he doesn't have to use it for it to serve as a record of the status of issues. Regards, Zooko
participants (6)
-
Ben Finney -
Cliff Wells -
Klaus Zimmermann -
Pete -
Phillip J. Eby -
zooko