Re: [Distutils] setuptools passing arguments/settings to dependencies' dependencies
At 05:54 AM 5/7/2010 -0700, Kent wrote:
Does this still work if you run "easy_install ." instead of
"setup.py install"?
No, when I try this it attempts to download from the internet: "Downloading http://pypi.python.org/packages/source/P/Paste/Paste-1.7.3.1.tar.gz#md5=743d..."
while processing a dependency's dependency.
This happens despite the fact that I have this section in setup.cfg:
========================================= [easy_install] index_url = ../../thirdparty
I thought you were using --find-links as of your last email? So, I'm a little confused as to what scenario you're actually trying to fix. The next thing I would do is verify that the config file is actually being read, by setting the DISTUTILS_DEBUG environment variable to "yes", and then running "easy_install -v ." and observing the output.
Like I mentioned, I do have a workaround in place that is working for me. I post it here in case it may help others:
Please don't do that, and *nobody else should do it either*. You are going to ERASE people's personal distutils configuration files with that code! (You're not even checking to see if it exists first!) I would be *extremely* upset if I tried to install your package and it erased one of my .pydistutils.cfg files, with all their command aliases, wikiup configs, path customizations, etc. (Of course, when run under easy_install control, the sandbox should catch your setup script's antics and terminate it with an error without harming any files, but still... it's pure evil and the sort of thing that should NEVER, ever be done in a setup.py, on pain of banishment from PyPI.)
I thought you were using --find-links as of your last email? So, I'm a little confused as to what scenario you're actually trying to fix.
Yes, I mentioned, I *used* to be using --find-links, but I determined this is not what I really wanted (though it did work in version c9).
The next thing I would do is verify that the config file is actually being read, by setting the DISTUTILS_DEBUG environment variable to "yes", and then running "easy_install -v ." and observing the output.
I know the answer is "absolutely, yes" for top level dependencies, as web access is blocked due to the setup.cfg line. Just not for the dependencies' dependencies. When I get a chance and I am in the office, I would gladly post the results of the debug.
Like I mentioned, I do have a workaround in place that is working for me. I post it here in case it may help others:
Please don't do that, and *nobody else should do it either*. You are going to ERASE people's personal distutils configuration files with that code! (You're not even checking to see if it exists first!)
Thank you for posting the disclaimer before use. I am exactly aware it is deleting that file and for my project that is completely acceptable, so I did not do anything fancy like backing it up. I'd hope anyone using this would have an understanding of python enough to see what is going on and modify it to meet their needs, but that is a poor assumption so a disclaimer would have been better; thank you for posting such.
I would be *extremely* upset if I tried to install your package and it erased one of my .pydistutils.cfg files, with all their command aliases, wikiup configs, path customizations, etc.
(Again, for my project this is a non-issue, as our company installs our software on our client's VMs - we are in complete control of the environment, so the "pure evil", as you called it isn't at all - it fixes an issue for us.)
(Of course, when run under easy_install control, the sandbox should catch your setup script's antics and terminate it with an error without harming any files, but still... it's pure evil and the sort of thing that should NEVER, ever be done in a setup.py, on pain of banishment from PyPI.)
I can only hope you are saying that tongue-in-cheek, because it is a bit harsh (how did I possibly offend you?). My workaround is acceptable for me, (not for a cheese shop project, I agree), but may help others in my circumstances. setuptools is very useful software even for projects that are never going to be publicly released. Thanks again for your help thus far.
The next thing I would do is verify that the config file is actually being read, by setting the DISTUTILS_DEBUG environment variable to "yes", and then running "easy_install -v ." and observing the output.
I've pasted the output here: http://pastebin.com/1it5XPAL Hopefully you can make sense of the output. Thanks again.
At 01:36 PM 5/7/2010 -0400, Kent Bower wrote:
Like I mentioned, I do have a workaround in place that is working for me. I post it here in case it may help others:
Please don't do that, and *nobody else should do it either*. You are going to ERASE people's personal distutils configuration files with that code! (You're not even checking to see if it exists first!)
Thank you for posting the disclaimer before use. I am exactly aware it is deleting that file and for my project that is completely acceptable, so I did not do anything fancy like backing it up. I'd hope anyone using this would have an understanding of python enough to see what is going on and modify it to meet their needs, but that is a poor assumption so a disclaimer would have been better; thank you for posting such.
It's not the part where it deletes *their* file that I'm worried about -- it's the people who might post such a thing to PyPI and delete random strangers' configuration files, in response to running, say, "setup.py --help". (Which your code will do.)
I would be *extremely* upset if I tried to install your package and it erased one of my .pydistutils.cfg files, with all their command aliases, wikiup configs, path customizations, etc.
(Again, for my project this is a non-issue, as our company installs our software on our client's VMs.)
Right - as long as you never distribute your project to *anyone else*. Ever.
(Of course, when run under easy_install control, the sandbox should catch your setup script's antics and terminate it with an error without harming any files, but still... it's pure evil and the sort of thing that should NEVER, ever be done in a setup.py, on pain of banishment from PyPI.) Hard to tell if you are saying that tongue-in-cheek, but if not, now you're being a bit harsh. I am working around a problem that is a deficit in the tool (as far as I can determine). In my circumstances what I am doing is perfectly acceptable and work-around of this nature are not "pure evil", it works and it may help someone else with exactly my issue, though it was rather amusing to read, just that I couldn't tell how serious you were being.
I think perhaps you're misunderstanding me. What I'm saying is that anybody who knowingly uploads code like that to PyPI does indeed need to be banned until they get the concept of not unconditionally deleting users' configuration files in a setup.py. It's precisely this sort of behavior that easy_install's sandboxing facility was added to prevent. In any case, what I don't get is, if you're distributing this only to your customers' VMs, why not just do one of the following: 1. Put the necessary settings in lib/distutils/distutils.cfg (configure globally for the VM, in other words) 2. Put them in ~/.pydistutils.cfg to start with, and leave 'em there 3. Specify them explicitly on the command line (I assume you're not having the users manually run setup.py install or easy_install, are you?) Don't get me wrong; if this behavior really changed between c9 and c11, I would like to understand it myself. What's weird is, it seems to me that it should have either always worked or always not worked. Although, it's possible that the way it was "working" in c9 was actually a side-effect of a bug I know I fixed by c11, wherein --find-links were sometimes being taken at too high a precedence -- although, in that case, it would've only manifested in cases where the installed version and the --find-links version were identical. The major thing that changed about dependency resolution by c11, is that when a dependency is built, sys.path and the working_set are put back the way they were before the build occurred, in order to prevent certain kinds of potential recursive loops and "poisoning" of the build environment. However, the main install command should never have been bothered by that in the first place, so I'm still having trouble wrapping my head around what you're saying. I'll probably have to dig into the code more before I can get a clue on this one.
On 5/7/2010 3:12 PM, P.J. Eby wrote:
It's not the part where it deletes *their* file that I'm worried about -- it's the people who might post such a thing to PyPI and delete random strangers' configuration files, in response to running, say, "setup.py --help". (Which your code will do.)
Ok, now we understand one another. This setuptools is very useful software even for cases where one has no intention of publicly releasing to PyPI. I failed to communicate this is the case for me....
I would be *extremely* upset if I tried to install your package and it erased one of my .pydistutils.cfg files, with all their command aliases, wikiup configs, path customizations, etc.
(Again, for my project this is a non-issue, as our company installs our software on our client's VMs.)
Right - as long as you never distribute your project to *anyone else*. Ever.
(Of course, when run under easy_install control, the sandbox should catch your setup script's antics and terminate it with an error without harming any files, but still... it's pure evil and the sort of thing that should NEVER, ever be done in a setup.py, on pain of banishment from PyPI.) Hard to tell if you are saying that tongue-in-cheek, but if not, now you're being a bit harsh. I am working around a problem that is a deficit in the tool (as far as I can determine). In my circumstances what I am doing is perfectly acceptable and work-around of this nature are not "pure evil", it works and it may help someone else with exactly my issue, though it was rather amusing to read, just that I couldn't tell how serious you were being.
I think perhaps you're misunderstanding me. What I'm saying is that anybody who knowingly uploads code like that to PyPI does indeed need to be banned until they get the concept of not unconditionally deleting users' configuration files in a setup.py.
It's precisely this sort of behavior that easy_install's sandboxing facility was added to prevent.
I was misunderstanding you, in fact, and I wholeheartedly agree, that would be evil (well, I don't know about "evil", but very wrong)... we are on the same page now.
In any case, what I don't get is, if you're distributing this only to your customers' VMs, why not just do one of the following:
1. Put the necessary settings in lib/distutils/distutils.cfg (configure globally for the VM, in other words) 2. Put them in ~/.pydistutils.cfg to start with, and leave 'em there 3. Specify them explicitly on the command line (I assume you're not having the users manually run setup.py install or easy_install, are you?)
#3 is exactly what I was trying to do... it doesn't work as of c11. #1 or #2 were my next ideas, but I really want to support the installation on these machines in more than one directory so that, once in production, it is trivial to revert to a previous release simply by running from the old release directory. Because of this, I cannot put an absolute path in lib/distutils/distutils.cfg or ~/.pydistutils.cfg because it changes relative to the thirdparty/ directory, which is relative to the location of setup.py.
Don't get me wrong; if this behavior really changed between c9 and c11, I would like to understand it myself. What's weird is, it seems to me that it should have either always worked or always not worked. Although, it's possible that the way it was "working" in c9 was actually a side-effect of a bug I know I fixed by c11, wherein --find-links were sometimes being taken at too high a precedence -- although, in that case, it would've only manifested in cases where the installed version and the --find-links version were identical.
The major thing that changed about dependency resolution by c11, is that when a dependency is built, sys.path and the working_set are put back the way they were before the build occurred, in order to prevent certain kinds of potential recursive loops and "poisoning" of the build environment. However, the main install command should never have been bothered by that in the first place, so I'm still having trouble wrapping my head around what you're saying. I'll probably have to dig into the code more before I can get a clue on this one.
participants (3)
-
Kent -
Kent Bower
-
P.J. Eby