Odd limitation in packaging/distutils2
This seems odd... PS D:\Data\PyPI> D:\Data\cpython\PCbuild\python -m packaging.run install nose installing third-party projects from an uninstalled Python is not supported And yet after downloading the nose source... PS D:\Data\PyPI> bsdtar xzf D:\Downloads\Python\nose-1.1.2.tar.gz PS D:\Data\PyPI> cd .\nose-1.1.2 PS D:\Data\PyPI\nose-1.1.2> D:\Data\cpython\PCbuild\python -m packaging.run install Installing from source directory: 'D:\\Data\\PyPI\\nose-1.1.2' Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.14.tar.gz Extracting in c:\users\gustav\appdata\local\temp\tmpecso_w Now working in c:\users\gustav\appdata\local\temp\tmpecso_w\distribute-0.6.14 Building a Distribute egg in D:\Data\PyPI\nose-1.1.2 warning: no files found matching 'Makefile' under directory 'docs' warning: no files found matching 'indexsidebar.html' under directory 'docs' creating build creating build\src ... etc (The build actually failed, but for reasons unrelated to the fact that it's an uninstalled Python, as far as I can see). Why not allow installing direct from PyPI? Paul.
Hi Paul,
PS D:\Data\PyPI> D:\Data\cpython\PCbuild\python -m packaging.run install nose installing third-party projects from an uninstalled Python is not supported There is a comment in the source to explain that:
# Python would try to install into the site-packages directory under # $PREFIX, but when running from an uninstalled code checkout we don't # want to create directories under the installation root (Original bug report: #12246)
And yet after downloading the nose source... [snip] Ah, I need to reopen the bug. The patch edited d2.install but forgot to touch the install_commands.
Regards
On 2 November 2011 14:18, Éric Araujo <merwok@netwok.org> wrote:
Hi Paul,
PS D:\Data\PyPI> D:\Data\cpython\PCbuild\python -m packaging.run install nose installing third-party projects from an uninstalled Python is not supported There is a comment in the source to explain that:
# Python would try to install into the site-packages directory under # $PREFIX, but when running from an uninstalled code checkout we don't # want to create directories under the installation root
(Original bug report: #12246)
And yet after downloading the nose source... [snip] Ah, I need to reopen the bug. The patch edited d2.install but forgot to touch the install_commands.
I'm still a bit confused. This is on Windows, and maybe things are different there, but installing seems to just work for me (except for the case where things are to be downloaded). What problem should I look for? The reason I care is that installing packages into a dev build is really useful for testing packaging changes. I'm not actually sure how to turn a dev build into an installed build on Windows... Paul.
Hi,
On 2 November 2011 14:18, Éric Araujo <merwok@netwok.org> wrote: I'm still a bit confused. This is on Windows, and maybe things are different there, but installing seems to just work for me (except for the case where things are to be downloaded). What problem should I look for?
On UNIX, the sysconfig module exposes information from the configure script, such as the install prefix. In an uninstalled built checkout, this install prefix defaults to /usr/local, so trying to install a distribution would create /usr/local/lib/python3.3/site-packages, which would be incorrect and polluting. That’s the problem I wanted to prevent. Can you define what you mean with “just work”? Does it install things into $checkout/Lib/site-packages?
The reason I care is that installing packages into a dev build is really useful for testing packaging changes. I'm not actually sure how to turn a dev build into an installed build on Windows...
http://docs.python.org/devguide/#quick-start : “On Windows, load the project file PCbuild\pcbuild.sln in Visual Studio, select Debug, and Build -> Build Solution;” Cheers
On 2 November 2011 15:40, Éric Araujo <merwok@netwok.org> wrote:
Can you define what you mean with “just work”? Does it install things into $checkout/Lib/site-packages?
Seems to. Certainly does for my own packages.
The reason I care is that installing packages into a dev build is really useful for testing packaging changes. I'm not actually sure how to turn a dev build into an installed build on Windows...
http://docs.python.org/devguide/#quick-start : “On Windows, load the project file PCbuild\pcbuild.sln in Visual Studio, select Debug, and Build -> Build Solution;”
Sorry, I wasn't clear. That I know. But that produces what I assume counts as a "non-installed" build (with python.exe in the PCBuild directory). Looking at is_python_build, it's just checking for the existence of Modules\Setup.*, so I suppose I could just rename or delete those files. It all seems a bit arbitrary, though. On a non-installed Python build (i.e., python built in a checkout) on Windows, purelib and site-packages look fine: PS D:\Data> D:\Data\cpython\PCbuild\python.exe Python 3.3.0a0 (default, Oct 28 2011, 19:56:45) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information.
import sysconfig, os sysconfig.get_path('purelib') 'D:\\Data\\cpython\\Lib\\site-packages' os.path.isdir(sysconfig.get_path('purelib')) True
It looks like this might be a Unix-only issue - if so, can I suggest that the restriction be removed on Windows, as it *is* useful to be able to install packages in a non-installed Python build. I'd argue that on Unix, get_path('purelib') should find the right site-packages even for an uninstalled build. But I've no idea if that's too hard, or is somehow otherwise an issue. And I don't need the functionality myself, so I'll leave that one for the Unix users to decide. Paul
Hi,
Can you define what you mean with “just work”? Does it install things into $checkout/Lib/site-packages? Seems to. Certainly does for my own packages.
Oh, nice. The site-packages directory is not in .hgignore, so I think this works by a happy accident.
The reason I care is that installing packages into a dev build is really useful for testing packaging changes.
Agreed! With virtual environments coming to the standard library, I think the ability to install projects from an uninstalled Python will become even more useful. I will reopen the bug and try to fix sysconfig on UNIX (#12141, #6087) to remove the limitation for all OSes.
I'm not actually sure how to turn a dev build into an installed build on Windows... http://docs.python.org/devguide/#quick-start : “On Windows, load the project file PCbuild\pcbuild.sln in Visual Studio, select Debug, and Build -> Build Solution;”
Sorry, I wasn't clear. That I know. But that produces what I assume counts as a "non-installed" build (with python.exe in the PCBuild directory).
It’s me, I confused building and installing. *puts stupid hat on* I don’t know if Visual Studio can install a project, or if you have to turn it into a installer somehow first and then click it. You could ask on IRC or python-dev.
Looking at is_python_build, it's just checking for the existence of Modules\Setup.*, so I suppose I could just rename or delete those files. It all seems a bit arbitrary, though.
I wouldn’t do that: It would trick Python into thinking it’s installed without actually doing all the things that make it installed. Regards
On 3 November 2011 16:45, Éric Araujo <merwok@netwok.org> wrote:
I wouldn’t do that: It would trick Python into thinking it’s installed without actually doing all the things that make it installed.
Agreed, it's not good. It's just that I don't know (on Windows) what things *do* make a Python build "installed". I'll see if I can find a definitive statement from somewhere... Paul.
On 4/11/2011 3:53 AM, Paul Moore wrote:
On 3 November 2011 16:45, Éric Araujo<merwok@netwok.org> wrote:
I wouldn’t do that: It would trick Python into thinking it’s installed without actually doing all the things that make it installed.
Agreed, it's not good. It's just that I don't know (on Windows) what things *do* make a Python build "installed". I'll see if I can find a definitive statement from somewhere...
There really isn't anything that makes it "installed". A built Python works completely fine just where it is (and indeed, will work just fine if moved somewhere else). The closest thing to being "installed" on windows is an "InstalledPath" (or something :) registry key for the version, but that is only necessary in a limited number of contexts - when some executable other than the in-place python[w].exe needs to know where things are. I think the original report is certainly a bug on Windows (and seems mis-guided on other platforms, but each to their own :) Mark
On 4 November 2011 12:11, Mark Hammond <skippy.hammond@gmail.com> wrote:
There really isn't anything that makes it "installed". A built Python works completely fine just where it is (and indeed, will work just fine if moved somewhere else). The closest thing to being "installed" on windows is an "InstalledPath" (or something :) registry key for the version, but that is only necessary in a limited number of contexts - when some executable other than the in-place python[w].exe needs to know where things are.
So that would imply that sysconfig.is_python_build is basically meaningless on Windows. So at a minimum, it should always return False on Windows. Actually, it's not part of the public API of sysconfig (it's not in __all__) and it's only used in one place within sysconfig.py, so maybe it should be removed and the check done inline in that one place... Paul.
Hi, Le 04/11/2011 13:11, Mark Hammond a écrit :
On 4/11/2011 3:53 AM, Paul Moore wrote:
Agreed, it's not good. It's just that I don't know (on Windows) what things *do* make a Python build "installed". I'll see if I can find a definitive statement from somewhere...
There really isn't anything that makes it "installed". A built Python works completely fine just where it is (and indeed, will work just fine if moved somewhere else). The closest thing to being "installed" on windows is an "InstalledPath" (or something :) registry key for the version, but that is only necessary in a limited number of contexts - when some executable other than the in-place python[w].exe needs to know where things are.
Ah, okay. The situation is different on UNIX: Installing Python typically writes the file in read-only mode (one consequence: users have to call setup.py --user, use virtualenv or switch to the superuser to install things), and does not just copy the build dir (so missing data files in the Makefile cause bugs that can only be seen when running the test suite or other code from an installed Python).
I think the original report is certainly a bug on Windows (and seems mis-guided on other platforms, but each to their own :)
I’ve already said that I want to remove the limitation for all OSes :) Cheers
In article <4EB41FF3.1080304@netwok.org>, Éric Araujo <merwok@netwok.org> wrote:
Le 04/11/2011 13:11, Mark Hammond a écrit :
On 4/11/2011 3:53 AM, Paul Moore wrote:
Agreed, it's not good. It's just that I don't know (on Windows) what things *do* make a Python build "installed". I'll see if I can find a definitive statement from somewhere...
There really isn't anything that makes it "installed". A built Python works completely fine just where it is (and indeed, will work just fine if moved somewhere else). The closest thing to being "installed" on windows is an "InstalledPath" (or something :) registry key for the version, but that is only necessary in a limited number of contexts - when some executable other than the in-place python[w].exe needs to know where things are.
Ah, okay. The situation is different on UNIX: Installing Python typically writes the file in read-only mode (one consequence: users have to call setup.py --user, use virtualenv or switch to the superuser to install things), and does not just copy the build dir (so missing data files in the Makefile cause bugs that can only be seen when running the test suite or other code from an installed Python).
One other very important difference: if the build was configured as --enable-shared, the final installed path to the shared library is usually captured in the built python executable. On most Unixy platforms that means you *cannot* reliably run the python executable from the build directory without taking some steps to override the default dynamic library search path, i.e. usually LD_LIBRARY_PATH. In the best case, the interpreter fails to start; in the worst, you get deceptive behavior when the newly built interpreter executable actually runs with a previously installed shared library. This is also true for OS X framework builds, which are a special kind of shared-library builds. -- Ned Deily, nad@acm.org
participants (4)
-
Mark Hammond -
Ned Deily -
Paul Moore -
Éric Araujo