Re: [Distutils] reservations about pythonv
This alternative approach (with a symlink and config file) sounds pretty good to me as well after our discussion at the sprints. The downsides I see:
1. Cross-platform inconsistency. Windows virtualenvs would be isolated from system python binary upgrades, *nix would not (some, like Brandon, consider this isolation a feature, so maybe *nix should allow that option too?) On the whole this doesn't seem like a big issue.
2. There would be no way to set LD_LIBRARY_PATH to include the virtualenv prior to execution of python. To be honest, I don't understand why this is needed, as current virtualenv doesn't do it and I haven't seen it break anything (compiled modules work fine in a venv). But people with more experience seemed to think it was needed at our open space discussion last Saturday. Perhaps someone can clarify what specifically breaks without this?
I'll dive into this approach and see what I can achieve.
I'd also appreciate feedback from Tarek or others familiar with the new sysconfig module about my changes there. Something along those lines will I think be needed with either approach, but I'm almost wondering if a new "virtual" install scheme would be a better approach?
Carl
Sent from my Android phone
----- Reply message ----- From: "Vinay Sajip" vinay_sajip@yahoo.co.uk Date: Wed, Mar 16, 2011 6:13 pm Subject: [Distutils] reservations about pythonv To: Distutils-Sig@Python.Org
Barry Warsaw <barry <at> python.org> writes:
Here at the sprints we talked about several possible options, the details of which of course will have to be hashed out. There will also be cross platform considerations. I think on *nix at least, it would be nice if a symlink and configuration file were enough to trigger the virtualenv behavior.
I feel that this approach definitely has potential to be the most useful and practical. There's no reason for a separate executable if Python itself incorporates the virtual environment functionality, which is mostly about setting paths and environment variables.
For example, if I have a local 'python' symlinked to the real executable, with a pythonv.conf file next to it, the virtual environment would be enabled. The real Python binary would adjust its behavior in that case to know where the standard library was, but also use the locally installed packages. Anyway, that's how I'd *like* it to work on *nix, but it may have to work differently on Windows, and it may have to work differently if environment variables have to be set.
Even if on Windows you have to copy rather than symlink, that could just be the main Python executable, which is not prohibitively large since the core of Python is in pythonX.Y.dll.
Regards,
Vinay Sajip
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Hi all,
I've pushed this symlink/copy binary approach to the "pythonv2" branch at http://bitbucket.org/carljm/cpythonv. I was a bit shocked at how easy it turned out to be: just a few lines in site.py and the same changes to sysconfig.py as previously. No changes in C code needed at all. It was so easy I'm wondering what I must have missed, but everything seems to work well. Please try it and tell me what I missed!
To try it out, create a directory somewhere with a bin/ subdirectory, symlink or copy (it works either way) the python3 binary from an install of the pythonv2 branch into bin/, and create lib/python3.3/site-packages in the virtualenv. Create "pythonv.conf" in the root of the virtualenv - it can be empty if you want full isolation; if you want the system site-packages too you can add a [pythonv] section with "include-system-site=True" (it's a ConfigParser-style file). Start up your binary and check sys.path. You can download distribute and "python setup.py install" that into your virtualenv and then easy_install more stuff into your env.
At this point, unless the lack of LD_LIBRARY_PATH is actually a blocker, I can't see any reason not to go with this approach instead.
Carl
On Mar 16, 2011, at 10:33 PM, Carl Meyer wrote:
I've pushed this symlink/copy binary approach to the "pythonv2" branch at http://bitbucket.org/carljm/cpythonv. I was a bit shocked at how easy it turned out to be: just a few lines in site.py and the same changes to sysconfig.py as previously. No changes in C code needed at all. It was so easy I'm wondering what I must have missed, but everything seems to work well. Please try it and tell me what I missed!
Seems to work for me (on Linux at least :)! Really brilliant work, Carl.
To try it out, create a directory somewhere with a bin/ subdirectory, symlink or copy (it works either way) the python3 binary from an install of the pythonv2 branch into bin/, and create lib/python3.3/site-packages in the virtualenv. Create "pythonv.conf" in the root of the virtualenv - it can be empty if you want full isolation; if you want the system site-packages too you can add a [pythonv] section with "include-system-site=True" (it's a ConfigParser-style file). Start up your binary and check sys.path. You can download distribute and "python setup.py install" that into your virtualenv and then easy_install more stuff into your env.
At this point, unless the lack of LD_LIBRARY_PATH is actually a blocker, I can't see any reason not to go with this approach instead.
+1. Time for a PEP?
-Barry
On 03/16/2011 11:04 PM, Barry Warsaw wrote:
+1. Time for a PEP?
Working on a draft PEP. I'll push it to bitbucket to make collaboration easier - then the next step would be to present the draft on python-ideas, if I'm reading PEP 2 correctly?
Carl
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 03/17/2011 02:50 PM, Carl Meyer wrote:
On 03/16/2011 11:04 PM, Barry Warsaw wrote:
+1. Time for a PEP?
Working on a draft PEP. I'll push it to bitbucket to make collaboration easier - then the next step would be to present the draft on python-ideas, if I'm reading PEP 2 correctly?
Correct.
Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com
On Mar 17, 2011, at 02:50 PM, Carl Meyer wrote:
On 03/16/2011 11:04 PM, Barry Warsaw wrote:
+1. Time for a PEP?
Working on a draft PEP. I'll push it to bitbucket to make collaboration easier - then the next step would be to present the draft on python-ideas, if I'm reading PEP 2 correctly?
I dunno. I tend to think python-ideas are where ideas go to die. ;) I think we have general consensus that this is a good idea, and we just need to hash out the details. So I personally think python-dev is fine at this point.
(Has nothing to do with me not actually being subscribed to python-ideas. Nope, not at all. :)
-Barry
On 03/17/2011 03:16 PM, Barry Warsaw wrote:
I dunno. I tend to think python-ideas are where ideas go to die. ;) I think we have general consensus that this is a good idea, and we just need to hash out the details. So I personally think python-dev is fine at this point.
Seems like it can't hurt to at least ping python-ideas about it (and follow the documented process). I can be pretty clear that distutils-sig already thinks it's a good idea in principle, there's no reason we have to let it die in python-ideas. And who knows, someone there may have some good ideas...
Carl
Carl Meyer <carl <at> oddbird.net> writes:
work well. Please try it and tell me what I missed!
To try it out, create a directory somewhere with a bin/ subdirectory, symlink or copy (it works either way) the python3 binary from an install of the pythonv2 branch into bin/, and create lib/python3.3/site-packages
It seems to work with a symlink but not with a copy. With an empty pythonv.conf file in ~/projects/vptest:
vinay@eta-natty:~/projects/vptest/bin$ ln -s ~/tools/cpythonv/python vinay@eta-natty:~/projects/vptest/bin$ ./python Python 3.3a0 (pythonv2:e56f05883ceb, Mar 17 2011, 09:07:29) [GCC 4.5.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import sys sys.path
['', '/usr/local/lib/python33.zip', '/home/vinay/tools/cpythonv/Lib', '/home/vinay/tools/cpythonv/Lib/plat-linux2', '/home/vinay/tools/cpythonv/build/lib.linux-i686-3.3', '/home/vinay/projects/vptest/lib/python3.3/site-packages']
vinay@eta-natty:~/projects/vptest/bin$ rm python vinay@eta-natty:~/projects/vptest/bin$ cp ~/tools/cpythonv/python . vinay@eta-natty:~/projects/vptest/bin$ ./python Python 3.3a0 (pythonv2:e56f05883ceb, Mar 17 2011, 09:07:29) [GCC 4.5.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import sys sys.path
['', '/usr/local/lib/python33.zip', '/usr/local/lib/python3.3', '/usr/local/lib/python3.3/plat-linux2', '/usr/local/lib/python3.3/lib-dynload', '/usr/local/lib/python3.3/site-packages']
Still, very promising! Nice work.
Regards,
Vinay Sajip
Hi Vinay,
On 03/17/2011 06:25 AM, Vinay Sajip wrote:
It seems to work with a symlink but not with a copy. With an empty pythonv.conf file in ~/projects/vptest:
vinay@eta-natty:~/projects/vptest/bin$ ln -s ~/tools/cpythonv/python vinay@eta-natty:~/projects/vptest/bin$ ./python Python 3.3a0 (pythonv2:e56f05883ceb, Mar 17 2011, 09:07:29) [GCC 4.5.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import sys sys.path
['', '/usr/local/lib/python33.zip', '/home/vinay/tools/cpythonv/Lib', '/home/vinay/tools/cpythonv/Lib/plat-linux2', '/home/vinay/tools/cpythonv/build/lib.linux-i686-3.3', '/home/vinay/projects/vptest/lib/python3.3/site-packages']
vinay@eta-natty:~/projects/vptest/bin$ rm python vinay@eta-natty:~/projects/vptest/bin$ cp ~/tools/cpythonv/python . vinay@eta-natty:~/projects/vptest/bin$ ./python Python 3.3a0 (pythonv2:e56f05883ceb, Mar 17 2011, 09:07:29) [GCC 4.5.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import sys sys.path
['', '/usr/local/lib/python33.zip', '/usr/local/lib/python3.3', '/usr/local/lib/python3.3/plat-linux2', '/usr/local/lib/python3.3/lib-dynload', '/usr/local/lib/python3.3/site-packages']
This is odd, as the same setup works fine for me with a copied binary (I'm on Ubuntu 10.10). If you're willing to do any more debugging on this, here's what would be helpful:
1. Check the values of sys.executable, sys.prefix, and sys.exec_prefix. 2. Stick a pdb.set_trace() into site.py near the top of the virtualize() method and step through it, observing where something goes awry. In particular: does it find pythonv.conf? Does it modify PREFIXES?
Thanks for testing it out!
Carl
Carl Meyer <carl <at> oddbird.net> writes:
This is odd, as the same setup works fine for me with a copied binary (I'm on Ubuntu 10.10). If you're willing to do any more debugging on this, here's what would be helpful:
I think I know what the problem is: the python executable checks to see where it was run from. If it looks as if it was run from a source build, it looks for site.py in the Lib folder relative to the executable; otherwise it looks for site.py in sys.prefix/lib/pythonX.Y. I've copied ~/tools/cpythonv/python to ~/projects/vptest/bin, and here's what happens:
vinay@eta-natty:~/projects/vptest/bin$ ~/tools/cpythonv/python Python 3.3a0 (pythonv2:e56f05883ceb, Mar 17 2011, 09:07:29) [GCC 4.5.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import sys; sys.prefix
'/usr/local'
sys.path
['', '/usr/local/lib/python33.zip', '/home/vinay/tools/cpythonv/Lib', '/home/vinay/tools/cpythonv/Lib/plat-linux2', '/home/vinay/tools/cpythonv/build/lib.linux-i686-3.3', '/usr/local/lib/python3.3/site-packages']
vinay@eta-natty:~/projects/vptest/bin$ ./python Python 3.3a0 (pythonv2:e56f05883ceb, Mar 17 2011, 09:07:29) [GCC 4.5.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import sys; sys.prefix
'/usr/local'
sys.path
['', '/usr/local/lib/python33.zip', '/usr/local/lib/python3.3', '/usr/local/lib/python3.3/plat-linux2', '/usr/local/lib/python3.3/lib-dynload', '/usr/local/lib/python3.3/site-packages']
Notice how the sys.prefix is the same in both cases, but the path is different. Your site.py is never run with the copy, because it gets the site.py from /usr/local/lib/python3.3:
vinay@eta-natty:~/projects/vptest/bin$ ~/tools/cpythonv/python Python 3.3a0 (pythonv2:e56f05883ceb, Mar 17 2011, 09:07:29) [GCC 4.5.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import site; site.__file__
'/home/vinay/tools/cpythonv/Lib/site.py'
vinay@eta-natty:~/projects/vptest/bin$ ./python Python 3.3a0 (pythonv2:e56f05883ceb, Mar 17 2011, 09:07:29) [GCC 4.5.2] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import site; site.__file__
'/usr/local/lib/python3.3/site.py'
I suppose this won't be a problem when the actual Python 3.3 is installed - the virtualizing site.py will be in /usr/lib/python3.3 or /usr/local/lib/python3.3. On this system I have a local build of Python made from the official default branch, which is installed to /usr/local/lib since the system Python is 3.2 (Ubuntu 11.04 - Natty).
Regards,
Vinay Sajip
Vinay Sajip <vinay_sajip <at> yahoo.co.uk> writes:
On this system I have a local build of Python made from the official default branch, which is installed to /usr/local/lib since the system Python is 3.2 (Ubuntu 11.04 - Natty).
Sorry, I meant to say that the system Python is 2.7 (in /usr). I also have an official Ubuntu build of 3.2 in /usr, but the source build installs by default to /usr/local.
Regards,
Vinay Sajip
Hi Vinay,
Thanks for the additional digging in here. I think your analysis is right - it actually occurred to me yesterday that this could be the problem, and I filed a bug to track it here: https://bitbucket.org/carljm/cpythonv/issue/6/if-binary-is-copied-prefix-fin...
The issue is finding the initial sys.prefix and thus the standard library, which happens in the search_for_prefix function Modules/getpath.c. The algorithm used is this (presuming no PYTHONHOME):
1. Step up the tree from the (symlinks-dereferenced) binary location looking for anything that appears to be the standard library.
2. Fall back to the location hardcoded in the binary from the --prefix option to ./configure.
With a symlink it's fully reliable, because the symlink is dereferenced so the stdlib is found just as it normally would be.
With a copy of the binary, there are two situations where it can break:
A. The Python installation we are copying from has an incorrect hardcoded prefix in the binary, and is relying on the tree-stepping to find its standard library. In this case, the hardcoded prefix will be used because the tree-stepping won't find anything. I think this is the situation you are seeing: I think your binary is compiling with a prefix of "/usr/local", and so it falls back on that prefix when it can't find anything by tree-stepping up through /home. If you didn't actually have a Python 3.3 installation in /usr/local, it would break more loudly due to not finding any stdlib at all. I think you'll find that it works for you if you compile with an explicit --prefix and then actually install to that prefix, then copy that binary and make an env with it.
B. There is _another_ Python standard library (from the same version of Python, on *nix - the Windows stdlib isn't in a version-specific directory) up the hierarchy from the location of our virtualenv. In this case, tree-stepping might find the wrong standard library and use it. I think this case should be quite rare. It's most problematic on Windows, but that's also where it shouldn't really happen, since Python is installed self-contained and I can't imagine why you'd make a virtualenv for one Python version _inside_ the installation directory of another Python version. In Linux I guess somebody might try to make a virtualenv, with a copied binary, somewhere in /usr/local, using some Python _other_ than the one installed in /usr, but with the same major/minor version. A very edge case, but possible.
So, possibilities I see for addressing this:
1. Decide that in real cases of real Python installations, it's so unlikely to happen that we won't worry about it. I think it's possible that this is acceptable; the biggest practical problem is likely to be people trying to test this out during PEP review from a not-installed checkout, just like you did. We'd have to be careful to instruct people that it doesn't work that way, and might also want to add a check in the env-creation script to verify that the created env works properly, and if it doesn't give them some clue why not.
2. Decide that we just don't support copied binaries, only symlinked ones. Apparently (I am Windows-ignorant) recent Windows versions do support symlinks? So this might only involve dropping support for old Windows'? How important is it for a new feature like this to fully support all operating systems that Python supports? We could also not expose the copy-binary option to the user, but fall back to it if we have no symlinks; which ends up being option (1) but trying to narrow even more the potential breakage cases.
3. The fully-reliable fix would be to somehow give the copied binary a hint where to find the right standard library, and this would involve adding something to the algorithm in getpath.c. The hint could take the form of a key in the config file, but I'd really like to avoid fully parsing the config-file in C and then again in Python later on. The hint could also be some kind of specially-formatted comment line at the top of the config file, which would require less C code to find and parse?
Any thoughts on this (or alternative solutions I haven't thought of) are most welcome.
Carl
At 02:44 PM 3/18/2011 -0400, Carl Meyer wrote:
Apparently (I am Windows-ignorant) recent Windows versions do support symlinks?
Technically, some Windows filesystems can support this. In practice, no user-visible tools actually support making or using them sanely, AFAIK.
So, I suggest promoting symlinks as the standard way, with binary-copying being a Windows-only workaround.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 03/18/2011 03:20 PM, P.J. Eby wrote:
At 02:44 PM 3/18/2011 -0400, Carl Meyer wrote:
Apparently (I am Windows-ignorant) recent Windows versions do support symlinks?
Technically, some Windows filesystems can support this. In practice, no user-visible tools actually support making or using them sanely, AFAIK.
So, I suggest promoting symlinks as the standard way, with binary-copying being a Windows-only workaround.
Could the config file contain an optional hint for finding the "right" stdlib in cases where the binary copy had been made? I realize that parsing a config file *without* the stdlib is painful: perhaps looking for a line starting with 'stdlib =' would be enough?
Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com
On 03/18/2011 05:24 PM, Tres Seaver wrote:
Could the config file contain an optional hint for finding the "right" stdlib in cases where the binary copy had been made? I realize that parsing a config file *without* the stdlib is painful: perhaps looking for a line starting with 'stdlib =' would be enough?
Yes, this was one possible solution I was considering. Two things bother me about it:
1. It seems... tempting but ill-advised to make it a "fake" ConfigParser option if in the place where I use it, I'm not actually parsing the file with full ConfigParser semantics. It might work, but then someone tries to get clever with a [DEFAULT] section, or interpolation, or who knows what else, and it breaks mysteriously. That's why I was leaning towards a specially-tagged comment at the top of the file. Ugly, but at least not pretending to be something it isn't.
2. I still have to duplicate in C and Python code the logic for finding the config file in the first place. Or I guess just move it to C and stash the location in sys or somewhere the Python code can get to it later. I'd been hoping to avoid this, but oh well...
Carl
- It seems... tempting but ill-advised to make it a "fake" ConfigParser
option if in the place where I use it, I'm not actually parsing the file with full ConfigParser semantics. It might work, but then someone tries to get clever with a [DEFAULT] section, or interpolation, or who knows what else, and it breaks mysteriously.
It’s still time to define the file format as RawConfigParser-style, without interpolation.
- I still have to duplicate in C and Python code the logic for finding
the config file in the first place. Or I guess just move it to C and stash the location in sys or somewhere the Python code can get to it later. I'd been hoping to avoid this, but oh well...
*hands a courage cookie* I have to say the proof of concept in pure Python was impressive. Too bad it’s not enough :)
Cheers
On 03/18/2011 07:22 PM, Éric Araujo wrote:
- It seems... tempting but ill-advised to make it a "fake" ConfigParser
option if in the place where I use it, I'm not actually parsing the file with full ConfigParser semantics. It might work, but then someone tries to get clever with a [DEFAULT] section, or interpolation, or who knows what else, and it breaks mysteriously.
It’s still time to define the file format as RawConfigParser-style, without interpolation.
Indeed. And I guess there's little to no reason why someone would try to define a default section, or use multiline syntax, or anything else like that. Perhaps this would be robust enough.
- I still have to duplicate in C and Python code the logic for finding
the config file in the first place. Or I guess just move it to C and stash the location in sys or somewhere the Python code can get to it later. I'd been hoping to avoid this, but oh well...
*hands a courage cookie* I have to say the proof of concept in pure Python was impressive. Too bad it’s not enough :)
Courage cookie accepted ;)
Carl
On Fri, Mar 18, 2011 at 6:35 PM, Carl Meyer carl@oddbird.net wrote:
On 03/18/2011 05:24 PM, Tres Seaver wrote:
Could the config file contain an optional hint for finding the "right" stdlib in cases where the binary copy had been made? I realize that parsing a config file *without* the stdlib is painful: perhaps looking for a line starting with 'stdlib =' would be enough?
Yes, this was one possible solution I was considering. Two things bother me about it:
- It seems... tempting but ill-advised to make it a "fake" ConfigParser
option if in the place where I use it, I'm not actually parsing the file with full ConfigParser semantics. It might work, but then someone tries to get clever with a [DEFAULT] section, or interpolation, or who knows what else, and it breaks mysteriously. That's why I was leaning towards a specially-tagged comment at the top of the file. Ugly, but at least not pretending to be something it isn't.
Don't call it ConfigParser format. Call it "ini" format, or whatever.
Jim
Hi Carl,
So, possibilities I see for addressing this:
- Decide that in real cases of real Python installations, it's so
unlikely to happen that we won't worry about it. I think it's possible that this is acceptable; the biggest practical problem is likely to be people trying to test this out during PEP review from a not-installed checkout, just like you did. We'd have to be careful to instruct people that it doesn't work that way, and might also want to add a check in the env-creation script to verify that the created env works properly, and if it doesn't give them some clue why not.
Yes, I think this is important to do, or at least to have -v output show which stdlib is being used.
- Decide that we just don't support copied binaries, only symlinked
ones. Apparently (I am Windows-ignorant) recent Windows versions do support symlinks? So this might only involve dropping support for old Windows'? How important is it for a new feature like this to fully support all operating systems that Python supports? We could also not expose the copy-binary option to the user, but fall back to it if we have no symlinks; which ends up being option (1) but trying to narrow even more the potential breakage cases.
Well, Windows XP is still pretty common and likely to remain so for some time. On Windows XP, symlinks don't work, but confusingly os.symlink exists on recent Pythons (certainly 3.2). I had to make a patch in my virtualenv fork to handle this in copyfile: on XP, os.symlink raises NotImplementedError because the underlying Windows API function is missing.
So, even if it's just for Windows, I think copying will need to be supported, but perhaps only from Pythons whose sys.prefix is correctly set.
- The fully-reliable fix would be to somehow give the copied binary a
hint where to find the right standard library, and this would involve adding something to the algorithm in getpath.c. The hint could take the form of a key in the config file, but I'd really like to avoid fully parsing the config-file in C and then again in Python later on. The hint could also be some kind of specially-formatted comment line at the top of the config file, which would require less C code to find and parse?
Any thoughts on this (or alternative solutions I haven't thought of) are most welcome.
I know that ConfigParser format files are the first thing one thinks of, but perhaps a simpler C-parseable format deserves further consideration.
Regards,
Vinay Sajip
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 03/18/2011 08:01 PM, Vinay Sajip wrote:
Hi Carl,
So, possibilities I see for addressing this:
- Decide that in real cases of real Python installations, it's so
unlikely to happen that we won't worry about it. I think it's possible that this is acceptable; the biggest practical problem is likely to be people trying to test this out during PEP review from a not-installed checkout, just like you did. We'd have to be careful to instruct people that it doesn't work that way, and might also want to add a check in the env-creation script to verify that the created env works properly, and if it doesn't give them some clue why not.
Yes, I think this is important to do, or at least to have -v output show which stdlib is being used.
- Decide that we just don't support copied binaries, only symlinked
ones. Apparently (I am Windows-ignorant) recent Windows versions do support symlinks? So this might only involve dropping support for old Windows'? How important is it for a new feature like this to fully support all operating systems that Python supports? We could also not expose the copy-binary option to the user, but fall back to it if we have no symlinks; which ends up being option (1) but trying to narrow even more the potential breakage cases.
Well, Windows XP is still pretty common and likely to remain so for some time. On Windows XP, symlinks don't work, but confusingly os.symlink exists on recent Pythons (certainly 3.2). I had to make a patch in my virtualenv fork to handle this in copyfile: on XP, os.symlink raises NotImplementedError because the underlying Windows API function is missing.
So, even if it's just for Windows, I think copying will need to be supported, but perhaps only from Pythons whose sys.prefix is correctly set.
- The fully-reliable fix would be to somehow give the copied binary a
hint where to find the right standard library, and this would involve adding something to the algorithm in getpath.c. The hint could take the form of a key in the config file, but I'd really like to avoid fully parsing the config-file in C and then again in Python later on. The hint could also be some kind of specially-formatted comment line at the top of the config file, which would require less C code to find and parse?
Any thoughts on this (or alternative solutions I haven't thought of) are most welcome.
I know that ConfigParser format files are the first thing one thinks of, but perhaps a simpler C-parseable format deserves further consideration.
One possibility is to make the config file format just a sequence of long-form command-line options, one per line, with the leading '--' stripped. This option would involve inventing a '--use-prefix' option (bikeshedders, start your sprayguns ;), but would make parsing it a mostly trivial exercise.
Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com
Carl Meyer <carl <at> oddbird.net> writes:
Thanks for the additional digging in here. I think your analysis is right - it actually occurred to me yesterday that this could be the problem, and I filed a bug to track it here:
- The fully-reliable fix would be to somehow give the copied binary a
hint where to find the right standard library, and this would involve adding something to the algorithm in getpath.c. The hint could take the form of a key in the config file, but I'd really like to avoid fully parsing the config-file in C and then again in Python later on. The hint could also be some kind of specially-formatted comment line at the top of the config file, which would require less C code to find and parse?
Any thoughts on this (or alternative solutions I haven't thought of) are most welcome.
Hi Carl,
I've been thinking about the C-level configuration some more (cpythonv issue #6). Suppose we have a virtual env at absolute path <testenv>, and in <testenv>/bin we have a copied, non-installed Python, obtained from absolute path <pythonsrc>. Say we provide the location of the source in the configuration, and that this is read in getpath.c. Logically, it should just replace the argv0_path value, and if the subsequent logic is unchanged, at the point that site.py's main() runs, you would have (on Linux i686):
sys.prefix, sys.exec_prefix would have values set from the configure script ('/usr/local' by default). sys.path would be ['/usr/local/lib/python33.zip', '<pythonsrc>/Lib', '<pythonsrc>/Lib/plat-linux2', '<pythonsrc>/build/lib.linux-i686-3.3']
which seems OK.
If a virtual environment is in absolute path <testenv>, then after the call to virtualize() in site.py, we would have sys.prefix and sys.exec_prefix both set to <testenv>. Since sys.executable still points to <testenv>/bin, the code in sysconfig.py appears not to be doing the right thing, e.g. sysconfig._PROJECT_BASE would be <testenv>/bin, which seems wrong.
So it seems as if site.py and/or sysconfig.py might need further changes, as well as changes to getpath.c. What do you think?
Regards,
Vinay Sajip
Hi Vinay,
On 03/29/2011 09:45 AM, Vinay Sajip wrote:
I've been thinking about the C-level configuration some more (cpythonv issue #6). Suppose we have a virtual env at absolute path <testenv>, and in <testenv>/bin we have a copied, non-installed Python, obtained from absolute path <pythonsrc>. Say we provide the location of the source in the configuration, and that this is read in getpath.c. Logically, it should just replace the argv0_path value, and if the subsequent logic is unchanged, at the point that site.py's main() runs, you would have (on Linux i686):
sys.prefix, sys.exec_prefix would have values set from the configure script ('/usr/local' by default). sys.path would be ['/usr/local/lib/python33.zip', '<pythonsrc>/Lib', '<pythonsrc>/Lib/plat-linux2', '<pythonsrc>/build/lib.linux-i686-3.3']
which seems OK.
If a virtual environment is in absolute path <testenv>, then after the call to virtualize() in site.py, we would have sys.prefix and sys.exec_prefix both set to <testenv>. Since sys.executable still points to <testenv>/bin, the code in sysconfig.py appears not to be doing the right thing, e.g. sysconfig._PROJECT_BASE would be <testenv>/bin, which seems wrong.
So it seems as if site.py and/or sysconfig.py might need further changes, as well as changes to getpath.c. What do you think?
In general, I think the "copied binary" case should be as similar as possible to the "symlinked binary" case. Python doesn't dereference symlinks in setting sys.executable, so in either case sys.executable will point to the virtual environment's binary, which I think is clearly what we want (think code using sys.executable to decide how to re-exec itself).
The changes I made to sysconfig thus far were the minimum necessary to be able to successfully install stuff in a pythonv virtualenv. On a brief glance, it looks to me like you are right; we'll need to modify the setting of _PROJECT_BASE in sysconfig.py to be virtual-aware (this is part -- maybe all -- of what's needed to fix issue #1).
Also, see issue #10 - I'm wavering on whether changing sys.prefix and sys.exec_prefix to point to the virtualenv is actually the right approach. We're already forced to modify sysconfig.py to be virtual-aware in order to avoid having to hackily copy/symlink bits of the stdlib and include directories and the Makefile and config.h and whatnot into the virtualenv. Once we've taken the step of making sysconfig virtual-aware, it seems to me that perhaps sys.prefix is better left pointing to the actual Python installation, and we add a sys.virtual_prefix or some such to tell site.py and sysconfig.py where "virtualized" stuff belongs.
There are a variety of possible meanings of sys.prefix, and the only one we really want to modify is "where should I install third-party packages?" Looking at sys.prefix references in the stdlib, I see them in trace, tkinter, and idlelib. It appears to me that in all of those cases, having sys.prefix pointing to the virtualenv would be wrong.
If we switch to sys.virtual_prefix and leave sys.prefix alone, the danger would be that some installer that's using sys.prefix directly, and ignoring sysconfig.py, would try to install stuff to the global Python installation. But I'm not aware of any such installer, and I think we could safely call any installer of Python code that ignores the layout schemes in sysconfig thoroughly broken anyway.
Your thoughts? (Or anyone else?)
Carl
P.S. The elephant in the room here is the fact that if we want pythonv to be backwards-compatible with distutils/distribute, which we do, we can't get away with just modifying sysconfig.py, we also have to modify the old distutils/sysconfig.py. In general distutils is frozen; I'm hoping that Tarek will let me get away with a few changes for pythonv if they aren't changing APIs at all, just modifying some install-scheme paths to be virtual-aware.
Carl Meyer <carl <at> oddbird.net> writes:
In general, I think the "copied binary" case should be as similar as possible to the "symlinked binary" case. Python doesn't dereference symlinks in setting sys.executable, so in either case sys.executable will point to the virtual environment's binary, which I think is clearly what we want (think code using sys.executable to decide how to re-exec itself).
I agree.
The changes I made to sysconfig thus far were the minimum necessary to be able to successfully install stuff in a pythonv virtualenv. On a brief glance, it looks to me like you are right; we'll need to modify the setting of _PROJECT_BASE in sysconfig.py to be virtual-aware (this is part -- maybe all -- of what's needed to fix issue #1).
Also, see issue #10 - I'm wavering on whether changing sys.prefix and sys.exec_prefix to point to the virtualenv is actually the right approach. We're already forced to modify sysconfig.py to be virtual-aware in order to avoid having to hackily copy/symlink bits of the stdlib and include directories and the Makefile and config.h and whatnot into the virtualenv. Once we've taken the step of making sysconfig virtual-aware, it seems to me that perhaps sys.prefix is better left pointing to the actual Python installation, and we add a sys.virtual_prefix or some such to tell site.py and sysconfig.py where "virtualized" stuff belongs.
This sounds better, too.
If we switch to sys.virtual_prefix and leave sys.prefix alone, the danger would be that some installer that's using sys.prefix directly, and ignoring sysconfig.py, would try to install stuff to the global Python installation. But I'm not aware of any such installer, and I think we could safely call any installer of Python code that ignores the layout schemes in sysconfig thoroughly broken anyway.
Your thoughts? (Or anyone else?)
What about the possibility of third-party code which uses sys.prefix/sys.exec_prefix, though? (I mean apart from the installer code.)
P.S. The elephant in the room here is the fact that if we want pythonv to be backwards-compatible with distutils/distribute, which we do, we can't get away with just modifying sysconfig.py, we also have to modify the old distutils/sysconfig.py. In general distutils is frozen; I'm hoping that Tarek will let me get away with a few changes for pythonv if they aren't changing APIs at all, just modifying some install-scheme paths to be virtual-aware.
Here's hoping :-)
On the question of a C-readable configuration file, I've started work on this on the assumption that it's a UTF8-encoded file which contains lines of the form key = value. Having thought about it, since this file is only for bootstrapping the virtualenv anyway, there's no strong case for it to be ConfigParser-compatible.
I've modified calculate_path in getpath.c to look for the file adjacent to the executable or one level up. If found, we look for a line "home = /path/to/source/of/copied/python" and, if found, replace argv0_path with the value, then let the existing code take its course. I've modified your virtualize() in site.py to cater for the new config file format: we default the prefix to the parent of the directory containing the executable (hence, the virtualenv root), then set sys.virtual_prefix to this prefix, and optionally disallow the system site-packages from the system path. For now (since [distutils.]sysconfig is not sys.virtual_prefix aware) I continue to set sys.prefix and sys.exec_prefix to sys.virtual_prefix.
So as not to cause confusion with existing config files you might have, I changed the config file name to 'env.cfg'. This can be changed to the appropriate bikeshed colour in two places - site.py and getpath.c.
Progress seems good. To test:
1. Clone the repo at https://bitbucket.org/vinay.sajip/pythonv 2. In that repo, run ./configure and make 3. Download the "pmv.py" script at https://gist.github.com/893507 4. Run pmv.py with the python you just built - see the header comment in that script.
Note that the pmv.py (Poor Man's virtualize.py) script deliberately runs with a copied rather than a symlinked Python.
Regards,
Vinay Sajip
Hi Vinay,
On 03/29/2011 07:00 PM, Vinay Sajip wrote:
If we switch to sys.virtual_prefix and leave sys.prefix alone, the danger would be that some installer that's using sys.prefix directly, and ignoring sysconfig.py, would try to install stuff to the global Python installation. But I'm not aware of any such installer, and I think we could safely call any installer of Python code that ignores the layout schemes in sysconfig thoroughly broken anyway.
Your thoughts? (Or anyone else?)
What about the possibility of third-party code which uses sys.prefix/sys.exec_prefix, though? (I mean apart from the installer code.)
Right. The question is, for third-party code using sys.prefix/sys.exec_prefix, which value would be the right one, the installation prefix or the virtual prefix? It depends on what they're using it for. I did a quick Google Code Search for "sys.prefix," and unfortunately right on the front page I can find examples where either approach we take would be "wrong."
The majority of uses I found are people copying code from trace.py which is intended to exclude the standard library and system-wide packages from coverage tracing. In this case, leaving sys.prefix pointing to the installation prefix would give the desired result. Also, for instance, here where matplotlib's setup looks for DLLs on Windows, the system prefix would be correct: http://bit.ly/eQSVYc
On the other hand, here scons uses sys.prefix to try to find site-packages: http://bit.ly/ik0PwJ If we leave sys.prefix pointing to the installation directory, this would leave scons unable to find its config file in the virtualenv's site-packages directory. Unless they fix their code to use the get_python_lib() function in sysconfig, which is what they ought to be doing instead of duplicating all that logic.
So... it seems to me that we're likely to break _some_ third-party code using sys.prefix regardless of what we do here. My instinct says adding sys.virtual_prefix and leaving sys.prefix alone is the better approach, but I'm not very firmly entrenched in that position.
On the question of a C-readable configuration file, I've started work on this on the assumption that it's a UTF8-encoded file which contains lines of the form key = value. Having thought about it, since this file is only for bootstrapping the virtualenv anyway, there's no strong case for it to be ConfigParser-compatible.
Agreed. I only used ConfigParser initially because that was easiest from Python code :-)
I've modified calculate_path in getpath.c to look for the file adjacent to the executable or one level up. If found, we look for a line "home = /path/to/source/of/copied/python" and, if found, replace argv0_path with the value, then let the existing code take its course. I've modified your virtualize() in site.py to cater for the new config file format: we default the prefix to the parent of the directory containing the executable (hence, the virtualenv root), then set sys.virtual_prefix to this prefix, and optionally disallow the system site-packages from the system path. For now (since [distutils.]sysconfig is not sys.virtual_prefix aware) I continue to set sys.prefix and sys.exec_prefix to sys.virtual_prefix.
This sounds good.
So as not to cause confusion with existing config files you might have, I changed the config file name to 'env.cfg'. This can be changed to the appropriate bikeshed colour in two places - site.py and getpath.c.
Progress seems good. To test:
- Clone the repo at https://bitbucket.org/vinay.sajip/pythonv
- In that repo, run ./configure and make
- Download the "pmv.py" script at https://gist.github.com/893507
- Run pmv.py with the python you just built - see the header comment in that
script.
Note that the pmv.py (Poor Man's virtualize.py) script deliberately runs with a copied rather than a symlinked Python.
Hmm, it doesn't seem to be working for me. sys.prefix and sys.exec_prefix are still /usr/local using the python binary in my "pmv", and sys.virtual_prefix is not set at all, despite env.cfg being present and looking as it should.
Oddly when I run the compiled python binary directly from the checkout, it does set sys.virtual_prefix to point one level above the checkout, despite there being no env.cfg in the vicinity at all.
You can see both of these things (in reverse order) here: http://paste.pocoo.org/show/362798/
I'll dig in a bit more and see if I can figure out what's happening.
Is there a reason you didn't base your changesets on mine (in the cpythonv repo), and instead apparently copied over a bunch of the changes manually as a diff? I can transplant or copy your work back, or try to do a full merge, but in the long run if we'll both be working on this it seems best if we are easily able to merge our work back and forth.
Carl
Carl Meyer <carl <at> oddbird.net> writes:
So... it seems to me that we're likely to break _some_ third-party code using sys.prefix regardless of what we do here. My instinct says adding sys.virtual_prefix and leaving sys.prefix alone is the better approach, but I'm not very firmly entrenched in that position.
That's problematic, of course (third-party breakage, I mean). It may be that despite using virtual_prefix being a cleaner approach, it may be necessary to alter sys.prefix to ensure compatibility, a la virtualenv.
Hmm, it doesn't seem to be working for me. sys.prefix and sys.exec_prefix are still /usr/local using the python binary in my "pmv", and sys.virtual_prefix is not set at all, despite env.cfg being present and looking as it should.
Oddly when I run the compiled python binary directly from the checkout, it does set sys.virtual_prefix to point one level above the checkout, despite there being no env.cfg in the vicinity at all.
You can see both of these things (in reverse order) here: http://paste.pocoo.org/show/362798/
I'll dig in a bit more and see if I can figure out what's happening.
It may be that the thing isn't working properly, but it's masked on my system because I have an installed 3.3 in /usr/local (from when I was doing my pip/virtualenv tests). This means that there is a Makefile in a location which would not be there if I'd never installed 3.3 using sudo make install.
However, the getpath.c stuff should work - to test this, insert an "import pdb; pdb.set_trace()" in site.main, and check sys.path. This should contain entries from the virtualenv. The problems are more likely to be in the Python code: the C code always sets sys.prefix and sys.exec_prefix to /usr/local (or whatever configure was invoked with) and the sys.prefix / sys.exec_prefix stuff is done in site.virtualize, as before.
I'll try to test this on a machine which never had 3.3 installed - it seems you can't uninstall Python using sudo make uninstall :-(
Is there a reason you didn't base your changesets on mine (in the cpythonv repo), and instead apparently copied over a bunch of the changes manually as a diff? I can transplant or copy your work back, or try to do a full merge, but in the long run if we'll both be working on this it seems best if we are easily able to merge our work back and forth.
Sorry - I got in a muddle because I've got too many clones around and ran into free disk space issues, and goofed somewhere. Don't bother to do a full merge for now, as I'm not sure if my code's working yet. Since I don't expect to change more than 4 files (getpath.c, site.py, distutils/sysconfig.py and sysconfig.py) I don't think we'll have too much trouble merging in due course, and if/when we can agree that my experiment in pythonv is worth taking further, I'll do the work of merging into cpythonv. For now, though, my changes might be treading on your toes, making merging needlessly annoying.
I'll try to test on a Python-clean machine in a day or two, will report back then.
Regards,
Vinay
On 03/30/2011 07:18 PM, Vinay Sajip wrote:
So... it seems to me that we're likely to break _some_ third-party code using sys.prefix regardless of what we do here. My instinct says adding sys.virtual_prefix and leaving sys.prefix alone is the better approach, but I'm not very firmly entrenched in that position.
That's problematic, of course (third-party breakage, I mean). It may be that despite using virtual_prefix being a cleaner approach, it may be necessary to alter sys.prefix to ensure compatibility, a la virtualenv.
Well, as I said, I think there will be compatibility problems either way. We are splitting site-packages from the base Python installation, and when third-party code assumes that site-packages and the rest of the base Python installation are both found relative to sys.prefix, neither choice we make can possibly be right.
Virtualenv tries to avoid this problem by copying or symlinking in enough stuff from the base Python install in order to convince third parties using sys.prefix that they really are looking at a full Python installation. But that approach is hacky and far from problem-free, and requires keeping up with various things that third-party code might try to look for under sys.prefix (there is at least one current open bug on virtualenv, Tkinter failing in a Windows virtualenv, that is the result of this problem). The logical conclusion of that approach, if you want full compatibility with third-party code, is to just copy the entire Python installation to the new location.
So three options as I see it:
1) Add sys.virtual_prefix (or perhaps sys.site_prefix would be a better name), and require third-party code to use that (or sysconfig) to find site-packages, if it wants to be virtual-compatible.
2) Repoint sys.prefix, and require third-party code to understand some new sys.system_prefix or sys.installed_prefix or equivalent to find the rest of the base Python installation.
3) Attempt the virtualenv approach of trying to fool third-party code into thinking the virtualenv IS a full Python installation, even though it's not.
I think #1 is far preferable to #2, and in the long run preferable to #3 even if it involves some short-term pain. But maybe I'm being too idealistic, and #3 is the pragmatic choice even though it's "wrong."
It may be that the thing isn't working properly, but it's masked on my system because I have an installed 3.3 in /usr/local (from when I was doing my pip/virtualenv tests). This means that there is a Makefile in a location which would not be there if I'd never installed 3.3 using sudo make install.
However, the getpath.c stuff should work - to test this, insert an "import pdb; pdb.set_trace()" in site.main, and check sys.path. This should contain entries from the virtualenv. The problems are more likely to be in the Python code: the C code always sets sys.prefix and sys.exec_prefix to /usr/local (or whatever configure was invoked with) and the sys.prefix / sys.exec_prefix stuff is done in site.virtualize, as before.
No, if I break early in site.py (or run python -S), sys.path doesn't contain any virtualenv paths.
Sorry - I got in a muddle because I've got too many clones around and ran into free disk space issues, and goofed somewhere. Don't bother to do a full merge for now, as I'm not sure if my code's working yet. Since I don't expect to change more than 4 files (getpath.c, site.py, distutils/sysconfig.py and sysconfig.py) I don't think we'll have too much trouble merging in due course, and if/when we can agree that my experiment in pythonv is worth taking further, I'll do the work of merging into cpythonv. For now, though, my changes might be treading on your toes, making merging needlessly annoying.
I'll try to test on a Python-clean machine in a day or two, will report back then.
Ok, thanks!
Carl
Carl Meyer <carl <at> oddbird.net> writes:
The logical conclusion of that approach, if you want full compatibility with third-party code, is to just copy the entire Python installation to the new location.
We do want that compatibility, don't we? Using a virtual environment should be a no-brainer, as virtualenv currently is (most of the time). Of course, the downside of the copying would be around 50MB per virtualenv on Windows, and much less where symlinking or hardlinking is available. Not ideal, of course, but disk space is cheap ;-)
So three options as I see it:
- Add sys.virtual_prefix (or perhaps sys.site_prefix would be a better
name), and require third-party code to use that (or sysconfig) to find site-packages, if it wants to be virtual-compatible.
But the success of virtualenv is because you can install just about any package in an environment, and packages don't have to do anything to be "virtual compatible".
- Repoint sys.prefix, and require third-party code to understand some
new sys.system_prefix or sys.installed_prefix or equivalent to find the rest of the base Python installation.
I think that the answer is for third party code not to use any prefix directly, but rely on an API such as sysconfig.get_path('purelib'). There's a distutils version of this - distutils.sysconfig.get_python_lib() - but this is brittle as per issue #6087.
- Attempt the virtualenv approach of trying to fool third-party code
into thinking the virtualenv IS a full Python installation, even though it's not.
I think #1 is far preferable to #2, and in the long run preferable to #3 even if it involves some short-term pain. But maybe I'm being too idealistic, and #3 is the pragmatic choice even though it's "wrong."
I'm afraid you may be right, at least until people stop using sys.prefix and sys.exec_prefix directly and call functions instead, which can operate differently in a virtual environment.
Regards,
Vinay Sajip
On 03/31/2011 04:12 AM, Vinay Sajip wrote:
Carl Meyer <carl <at> oddbird.net> writes:
The logical conclusion of that approach, if you want full compatibility with third-party code, is to just copy the entire Python installation to the new location.
We do want that compatibility, don't we? Using a virtual environment should be a no-brainer, as virtualenv currently is (most of the time). Of course, the downside of the copying would be around 50MB per virtualenv on Windows, and much less where symlinking or hardlinking is available. Not ideal, of course, but disk space is cheap ;-)
People tend to create a lot of virtualenvs. I don't think copying the entire Python installation is viable (it is slow, and it also would mean that security updates to the base standard library would not be reflected in the virtualenv). If copying the entire base Python install were an adequate solution, this entire virtual-python proposal would be moot - you can do that right now and it works fine.
- Add sys.virtual_prefix (or perhaps sys.site_prefix would be a better
name), and require third-party code to use that (or sysconfig) to find site-packages, if it wants to be virtual-compatible.
But the success of virtualenv is because you can install just about any package in an environment, and packages don't have to do anything to be "virtual compatible".
Sure. "virtual-compatible" was a poor choice of terms - in the long run, there should be no such concept. In the short run, we would be clarifying the semantics of sys.prefix and introducing a new sys.site_prefix. And in the short run, the effect of code that used sys.prefix contrary to those semantics would be failure in a virtualenv.
But, as I mentioned, the fact that easy_install itself relies on finding site-packages relative to sys.prefix forces the issue - easy_install (including existing versions of it) needs to just work.
- Repoint sys.prefix, and require third-party code to understand some
new sys.system_prefix or sys.installed_prefix or equivalent to find the rest of the base Python installation.
I think that the answer is for third party code not to use any prefix directly, but rely on an API such as sysconfig.get_path('purelib'). There's a distutils version of this - distutils.sysconfig.get_python_lib() - but this is brittle as per issue #6087.
Yes, of course. If all code did this, we wouldn't have this problem; we would just fix sysconfig.py and site.py and all would be well.
- Attempt the virtualenv approach of trying to fool third-party code
into thinking the virtualenv IS a full Python installation, even though it's not.
I think #1 is far preferable to #2, and in the long run preferable to #3 even if it involves some short-term pain. But maybe I'm being too idealistic, and #3 is the pragmatic choice even though it's "wrong."
I'm afraid you may be right, at least until people stop using sys.prefix and sys.exec_prefix directly and call functions instead, which can operate differently in a virtual environment.
Yup.
Carl
Carl Meyer <carl <at> oddbird.net> writes:
People tend to create a lot of virtualenvs. I don't think copying the entire Python installation is viable (it is slow, and it also would mean that security updates to the base standard library would not be reflected in the virtualenv).
You're probably right - I haven't actually tried it.
But, as I mentioned, the fact that easy_install itself relies on finding site-packages relative to sys.prefix forces the issue - easy_install (including existing versions of it) needs to just work.
Presumably this means that we can't use site_prefix in the short term, and just have it there to allow code to evolve compatibility to it. So everything else just uses site.prefix and site.exec_prefix as it does currently, and we achieve parity with virtualenv as far as compatibility is concerned. Internally, we use sys.installed_prefix and sys.installed_exec_prefix when we need to. We still benefit from the fact that no copying of the standard lib is needed, we're just using the stock site.py etc.
BTW you might find this script handy while testing:
https://gist.github.com/896831
Regards,
Vinay Sajip
At 02:14 PM 4/1/2011 -0400, Carl Meyer wrote:
But, as I mentioned, the fact that easy_install itself relies on finding site-packages relative to sys.prefix forces the issue - easy_install (including existing versions of it) needs to just work.
No, it doesn't, actually.
easy_install only uses sys.prefix to identify "site" directories without having to test them. Its install paths actually come from the distutils -- so as long as the distutils "install" command has the right defaults, easy_install will work just fine. (Assuming sys.executable is correct.)
What easy_install does if it's asked to install to a directory that:
1) doesn't look like a site directory (based on sys.prefix), and 2) isn't listed in --site-dirs on the command line or in the config files,
is that it runs a test (using sys.executable) to see if .pth files are actually supported in that directory.
So, if sys.prefix leads it to believe that the distutils-supplied target directory isn't really a "site" directory, it checks for itself whether it functions as a site directory.
In other words, easy_install will run a bit slower in such a scenario (due to the extra time taken to test), but still function correctly. To stop it taking the extra time for the test, an additional command line option or configuration file option can be given.
At 01:35 PM 3/30/2011 -0400, Carl Meyer wrote:
So... it seems to me that we're likely to break _some_ third-party code using sys.prefix regardless of what we do here. My instinct says adding sys.virtual_prefix and leaving sys.prefix alone is the better approach, but I'm not very firmly entrenched in that position.
Looking at it from a software distribution POV, I would say that the virtual prefix is what it should point to, since that means things won't get installed to the wrong place.
(Of course, a configuration option could be used to override it, if really necessary.)
On 03/30/2011 11:23 PM, P.J. Eby wrote:
At 01:35 PM 3/30/2011 -0400, Carl Meyer wrote:
So... it seems to me that we're likely to break _some_ third-party code using sys.prefix regardless of what we do here. My instinct says adding sys.virtual_prefix and leaving sys.prefix alone is the better approach, but I'm not very firmly entrenched in that position.
Looking at it from a software distribution POV, I would say that the virtual prefix is what it should point to, since that means things won't get installed to the wrong place.
Indeed. The issue is that from every point of view other than software distribution (code trying to find anything other than site-directories), sys.prefix pointing to the virtualenv is wrong. Unless, like virtualenv, we try to make it "just right enough" by copying/symlinking things into the virtualenv that otherwise wouldn't need to be there.
But this may be moot; I didn't realize until I checked just now that setuptools (well, easy_install) builds its own list of site-dirs based on sys.prefix. It doesn't look like that's used for installation, but it is used for pre-flight checking pth-capability and finding pth files. So if easy_install directly relies on site-directories always being sys.prefix-based, that probably forces the issue.
(Of course, a configuration option could be used to override it, if really necessary.)
This seems like the worst option of all - then third-party code really would have no idea at all what sys.prefix is supposed to mean, or what can be reliably found there.
Carl
Carl Meyer <carl <at> oddbird.net> writes:
Hmm, it doesn't seem to be working for me. sys.prefix and sys.exec_prefix are still /usr/local using the python binary in my "pmv", and sys.virtual_prefix is not set at all, despite env.cfg being present and looking as it should.
Oddly when I run the compiled python binary directly from the checkout, it does set sys.virtual_prefix to point one level above the checkout, despite there being no env.cfg in the vicinity at all.
You can see both of these things (in reverse order) here: http://paste.pocoo.org/show/362798/
After testing on a machine with no Python 3.3 installed, I get a similar failure to yours (invalid Python installation due to missing Makefile). However, rhe getpath.c code seems to be working as expected: see http://paste.pocoo.org/show/363159/ for example. The path has been set to reflect the source build where the python was copied from. Of course at this point virtualize() hasn't been called, so virtual_prefix etc. haven't been set, and prefixes are pointing to /usr/local.
I think the problem is occurring because there's incomplete support for running from a source build. I'm not sure what to do about this; I think proper support will need changes in multiple other places, which I haven't bottomed out yet but which includes distutils.sysconfig (see issue #6087). Since running from a source build does not set sys.prefix and sys.exec_prefix, either this needs to change, or there needs to be some additional concept like "sys.python_home" which is available to scripts, or perhaps something else. Of course fixing the source build issue is of use only for core developers, so I'm not sure what appetite there'd be for regularising things for this case.
Regards,
Vinay Sajip
Vinay Sajip <vinay_sajip <at> yahoo.co.uk> writes:
I think the problem is occurring because there's incomplete support for running from a source build.
To confirm this, I installed pythonv - and then running pmv.py produces the expected result - an env with a copied Python which allows you to install stuff into the env. I packaged the pythonv I used as a .deb using checkinstall, so that it's uninstallable:
http://www.red-dove.com/pythonv_3.3-1_i386.deb
and having installed it, I re-ran pmv and did some smoke testing, output is at
http://paste.pocoo.org/show/363186/
I know we're not done yet, but at least this shows that the getpath.c code seems to be working (because sys.path seems correct when running the env's copy of Python) and distribute_setup.py installs without errors, followed by easy_installing sample packages setuptools-git, ply, Pygments, Jinja2, SQLAlchemy and coverage ... all seemingly without problems.
Of course, this is still using sys.prefix/sys.exec_prefix.
Regards,
Vinay
On 03/31/2011 09:21 AM, Vinay Sajip wrote:
Vinay Sajip <vinay_sajip <at> yahoo.co.uk> writes:
I think the problem is occurring because there's incomplete support for running from a source build.
To confirm this, I installed pythonv - and then running pmv.py produces the expected result - an env with a copied Python which allows you to install stuff into the env. I packaged the pythonv I used as a .deb using checkinstall, so that it's uninstallable:
http://www.red-dove.com/pythonv_3.3-1_i386.deb
and having installed it, I re-ran pmv and did some smoke testing, output is at
http://paste.pocoo.org/show/363186/
I know we're not done yet, but at least this shows that the getpath.c code seems to be working (because sys.path seems correct when running the env's copy of Python) and distribute_setup.py installs without errors, followed by easy_installing sample packages setuptools-git, ply, Pygments, Jinja2, SQLAlchemy and coverage ... all seemingly without problems.
Of course, this is still using sys.prefix/sys.exec_prefix.
Looks great! I can confirm it works when installed. I'll look into the source-build issues.
Carl
Carl Meyer <carl <at> oddbird.net> writes:
Looks great! I can confirm it works when installed. I'll look into the source-build issues.
I've also updated pmv.py at
https://gist.github.com/893507
to be closer to what we want for virtualize.py in the stdlib, so that you can do
python3.3 -m virtualize /tmp/testenv
and have it work:
vinay@eta-natty:~/projects/scratch$ python3.3 -m virtualize -h usage: virtualize.py [-h] [--no-distribute] [--no-site-packages] [--clear] ENV_DIR [ENV_DIR ...]
positional arguments: ENV_DIR A directory to create the environment in.
optional arguments: -h, --help show this help message and exit --no-distribute Don't install Distribute in the virtual environment. --no-site-packages Don't give access to the global site-packages dir to the virtual environment. --clear Delete the environment directory if it already exists. If not specified and the directory exists, an error is raised. vinay@eta-natty:~/projects/scratch$ python3.3 -m virtualize /tmp/testenv Installing distribute .......................................................................... .......................................................................... .......................................................................... .......................................................................... .............................done. vinay@eta-natty:~/projects/scratch$
On Mar 16, 2011, at 07:19 PM, Carl Meyer wrote:
This alternative approach (with a symlink and config file) sounds pretty good to me as well after our discussion at the sprints. The downsides I see:
- Cross-platform inconsistency. Windows virtualenvs would be isolated from
system python binary upgrades, *nix would not (some, like Brandon, consider this isolation a feature, so maybe *nix should allow that option too?) On the whole this doesn't seem like a big issue.
Agreed. Windows users are used to doing Python upgrades themselves since their operating system doesn't help them. So if they have to copy binaries, they'll have to be aware that when they upgrade Python, their virtualenvs won't get automatically updated.
For *nix users, I agree that it might be a useful option to either copy binaries or use a symlink, though I think the later should be the default.
- There would be no way to set LD_LIBRARY_PATH to include the virtualenv
prior to execution of python. To be honest, I don't understand why this is needed, as current virtualenv doesn't do it and I haven't seen it break anything (compiled modules work fine in a venv). But people with more experience seemed to think it was needed at our open space discussion last Saturday. Perhaps someone can clarify what specifically breaks without this?
I was thinking about this too. I'm trying to recall a similar situation I had in a long-previous job where we had to re-exec to pick up $LD_LIBRARY_PATH (on Solaris IIRC). The issue there was that the Python binary *itself* needed $LD_LIBRARY_PATH set before it could find some shared libraries, so that envar had to essentially be set before `python` could run.
We've got a different situation here though. We're assuming `python` itself runs just fine, but that some shared library in the virtualenv is needed to import an extension module. So, why doesn't Python set $LD_LIBRARY_PATH in its own environment once it knows its running in a virtualenv? That way, any subsequent searches for shared libraries would find them in the virtualenv.
OTOH, I vaguely recall that while this *should* work, it actually doesn't because once the executable has started, $LD_LIBRARY_PATH isn't consulted again.
The above just means my memory is too faulty to of much use ;). I'll just echo Carl's request for specific cases where $LD_LIBRARY_PATH needs to be set.
Cheers, -Barry
On Wed, Mar 16, 2011 at 10:39 PM, Barry Warsaw barry@python.org wrote:
I vaguely recall that while this *should* work, it actually doesn't because once the executable has started, $LD_LIBRARY_PATH isn't consulted again.
I recall less vaguely, since we've had to deal with this problem more recently than your Solaris antics. :-) The loader consults LD_LIBRARY_PATH before main() is called, initializing it's own state, and doesn't pick up changes. This makes a re-exec necessary if the new value is different from the original (worth checking to avoid an exec).
-Fred
-- Fred L. Drake, Jr. <fdrake at acm.org> "A storm broke loose in my mind." --Albert Einstein
On Mar 17, 2011, at 08:36 AM, Fred Drake wrote:
On Wed, Mar 16, 2011 at 10:39 PM, Barry Warsaw barry@python.org wrote:
I vaguely recall that while this *should* work, it actually doesn't because once the executable has started, $LD_LIBRARY_PATH isn't consulted again.
I recall less vaguely, since we've had to deal with this problem more recently than your Solaris antics. :-) The loader consults LD_LIBRARY_PATH before main() is called, initializing it's own state, and doesn't pick up changes. This makes a re-exec necessary if the new value is different from the original (worth checking to avoid an exec).
Thanks for helping bolster my memory! I still think setting $LD_LIBRARY_PATH won't be necessary in the majority of cases, so generally no re-exec should happen. I can imagine that if it *were* necessary, an appropriate section in the pythonv.conf file would trigger it. E.g.:
[env] LD_LIBRARY_PATH: /path/to/add
[pythonv] re-exec: true
Cheers, -Barry
On 03/17/2011 10:56 AM, Barry Warsaw wrote:
Thanks for helping bolster my memory! I still think setting $LD_LIBRARY_PATH won't be necessary in the majority of cases, so generally no re-exec should happen. I can imagine that if it *were* necessary, an appropriate section in the pythonv.conf file would trigger it. E.g.:
[env] LD_LIBRARY_PATH: /path/to/add
[pythonv] re-exec: true
I also realized last night that if the need for LD_LIBRARY_PATH is as rare as it seems to be, people could just as well set it themselves before running stuff in their virtualenv. We could even have our shell "activate" script set it, so you'd only have to set it yourself if using the virtualenv's binary directly without "activate." I'm a bit tempted in this direction, as the whole auto-re-exec business feels like it introduces an entirely different level of magic. But if it would make life easier, it shouldn't be hard to do.
Since I'm finding and parsing pythonv.conf in site.py, would it be acceptable to do the exec there? Or are there good reasons for it to happen earlier in C code? (I'm a bit loath to have to find and parse pythonv.conf twice).
If we do this, I'd also appreciate (for the PEP) a more concrete real-life example of what you might try do in a a virtualenv that would require this.
Carl
On Mar 17, 2011, at 12:32 PM, Carl Meyer wrote:
I also realized last night that if the need for LD_LIBRARY_PATH is as rare as it seems to be, people could just as well set it themselves before running stuff in their virtualenv. We could even have our shell "activate" script set it, so you'd only have to set it yourself if using the virtualenv's binary directly without "activate."
+1. Keeps Python nice and simple and AFAICT, it should work because it'll set the environment variable before Python starts executing.
OTOH, if no actual use case for LD_LIBRARY_PATH gets raised, then even this won't be necessary.
-Barry
The above just means my memory is too faulty to of much use ;). I'll just echo Carl's request for specific cases where $LD_LIBRARY_PATH needs to be set.
Here is a case that might resemble what you are talking about:
Compile a C extension that requires a shared library that is not in the standard system path. To import it, LD_LIBRARY_PATH needs to be right.
This is not really different from what happens in a compiled language, except in one way: In C, I can compile it -static or I can give the full path to the .so file. Either results in a thing that works without LD_LIBRARY_PATH.
With distutils, you can't. It goes to great lengths to ensure that you can only compile a C extension with "cc ... -L/some/directory -lname" -- I can't find any way to make it do "cc ... /some/directory/libname.so"
So, the real problem here is that distutils uses "cc -L", but it demonstrates a case where LD_LIBRARY_PATH can be important to a python program even when Python itself can run without it.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi Mark,
On 03/18/2011 01:46 PM, Mark Sienkiewicz wrote:
Here is a case that might resemble what you are talking about:
Compile a C extension that requires a shared library that is not in the standard system path. To import it, LD_LIBRARY_PATH needs to be right.
But this required shared library that is not on the standard system path - - is it something that might have been installed, via Python (distutils/setuptools/d2/packaging), in the virtualenv, where if it had been installed in that same way in the system Python installation it would be on the standard system path?
If so, do you have any specific examples of installable Python projects that install a shared library that other ones might need in order to compile, so that I can test this case and make it work?
If not, then it seems this is a general Python packaging issue, not anything specific to this virtual-python proposal. In which case I don't have to care, at least not just now ;-)
This is not really different from what happens in a compiled language, except in one way: In C, I can compile it -static or I can give the full path to the .so file. Either results in a thing that works without LD_LIBRARY_PATH.
With distutils, you can't. It goes to great lengths to ensure that you can only compile a C extension with "cc ... -L/some/directory -lname" -- I can't find any way to make it do "cc ... /some/directory/libname.so"
So, the real problem here is that distutils uses "cc -L", but it demonstrates a case where LD_LIBRARY_PATH can be important to a python program even when Python itself can run without it.
Thanks!
Carl
Another variant of this case, one we're actually facing here at our company (Nexedi) right now, is when you need to compile extension modules with libraries that are newer than the ones in the system, and you don't have root access. The absense of LD_LIBRARY_PATH means a segfault on an arbitrary moment during execution, not just an unloadable library.
On Fri, Mar 18, 2011 at 20:00, Carl Meyer carl@meyerloewen.net wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi Mark,
On 03/18/2011 01:46 PM, Mark Sienkiewicz wrote:
Here is a case that might resemble what you are talking about:
Compile a C extension that requires a shared library that is not in the standard system path. To import it, LD_LIBRARY_PATH needs to be right.
But this required shared library that is not on the standard system path
- is it something that might have been installed, via Python
(distutils/setuptools/d2/packaging), in the virtualenv, where if it had been installed in that same way in the system Python installation it would be on the standard system path?
If so, do you have any specific examples of installable Python projects that install a shared library that other ones might need in order to compile, so that I can test this case and make it work?
If not, then it seems this is a general Python packaging issue, not anything specific to this virtual-python proposal. In which case I don't have to care, at least not just now ;-)
This is not really different from what happens in a compiled language, except in one way: In C, I can compile it -static or I can give the full path to the .so file. Either results in a thing that works without LD_LIBRARY_PATH.
With distutils, you can't. It goes to great lengths to ensure that you can only compile a C extension with "cc ... -L/some/directory -lname" -- I can't find any way to make it do "cc ... /some/directory/libname.so"
So, the real problem here is that distutils uses "cc -L", but it demonstrates a case where LD_LIBRARY_PATH can be important to a python program even when Python itself can run without it.
Thanks!
Carl -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk2Dq+MACgkQFRcxmeyPUXJbHwCfa1FpWif7nRJK90EARxxs8yGq bTMAnAxNj8eU3ohfRiCZyHOlNGIiL33T =Zmep -----END PGP SIGNATURE----- _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Hi Leonardo,
On 03/19/2011 08:18 PM, Leonardo Rochael Almeida wrote:
Another variant of this case, one we're actually facing here at our company (Nexedi) right now, is when you need to compile extension modules with libraries that are newer than the ones in the system, and you don't have root access. The absense of LD_LIBRARY_PATH means a segfault on an arbitrary moment during execution, not just an unloadable library.
Hmm. So my design goal for pythonv is that it "just work" to the same extent a real installation of Python would. IIUC in this case even if you were compiling your own full installation of Python, you would still need to manually set LD_LIBRARY_PATH to ensure that your newer shared libraries would be found before the system ones. So I don't think pythonv would change anything here.
To be clear, pythonv is not going to _prevent_ anyone from setting LD_LIBRARY_PATH before running the virtualized python. The open question is whether pythonv needs to automatically and transparently set LD_LIBRARY_PATH to point to some location in the virtual environment every time you use it, in order to work as well as a normal installation of Python.
Actually, the comparison to installing a custom-compiled Python to a non-standard location increases my sense that pythonv should not automatically add to LD_LIBRARY_PATH. If you "make install" Python to a non-standard location (on my Ubuntu system I have various versions of Python that I've compiled and installed in /opt), it doesn't take it upon itself to modify your system's library path accordingly, does it? So why should pythonv do any differently?
Carl
Hi Carl,
On Sun, Mar 20, 2011 at 04:06, Carl Meyer carl@oddbird.net wrote:
Hi Leonardo,
On 03/19/2011 08:18 PM, Leonardo Rochael Almeida wrote:
Another variant of this case, one we're actually facing here at our company (Nexedi) right now, is when you need to compile extension modules with libraries that are newer than the ones in the system, and you don't have root access. The absense of LD_LIBRARY_PATH means a segfault on an arbitrary moment during execution, not just an unloadable library.
[...]
Actually, the comparison to installing a custom-compiled Python to a non-standard location increases my sense that pythonv should not automatically add to LD_LIBRARY_PATH. If you "make install" Python to a non-standard location (on my Ubuntu system I have various versions of Python that I've compiled and installed in /opt), it doesn't take it upon itself to modify your system's library path accordingly, does it? So why should pythonv do any differently?
It's likely that pythonv shoudn't do anything differently, at least from the POV of the Python binary. I just wanted to mention a use-case for needing LD_LIBRARY_PATH set while running Python.
This being distutils-sig, though, maybe we could talk about whether supporting scripts or generated scripts could do something about it. For example, the 'activate' script, or, in buildout, pip, etc. case, shebang lines that included an 'env' with the correct LD_LIBRARY_PATH, like:
#!/usr/bin/env LD_LIBRARY_PATH=/my/custom/lib /my/custom/bin/python
Cheers,
Leo
On Sun, Mar 20, 2011 at 1:48 PM, Leonardo Rochael Almeida leorochael@gmail.com wrote:
It's likely that pythonv shoudn't do anything differently, at least from the POV of the Python binary. I just wanted to mention a use-case for needing LD_LIBRARY_PATH set while running Python.
I'd agree that for the use-cases I've dealt with, it's not clear that pythonv in particular should be responsible for handling LD_LIBRARY_PATH; it would at times be convenient, but not essential. Existing workarounds would still apply, and in some cases would still be necessary.
There are two primary changes we've wanted to make when the Python process is started:
- setting environment variables that have to be set for the dynamic loaded (LD_LIBRARY_PATH is the most common), and
- running as a particular user.
Both of these would require restarting Python to get set up, and neither is part of getting the basic "virtual" environment set up. Getting environment changes made & restarting the interpreter wouldn't solve both of these, and the run-as-user behavior is definitely inappropriate for a basic pythonv behavior.
-Fred
-- Fred L. Drake, Jr. <fdrake at acm.org> "A storm broke loose in my mind." --Albert Einstein
On Mar 20, 2011, at 01:18 AM, Leonardo Rochael Almeida wrote:
Another variant of this case, one we're actually facing here at our company (Nexedi) right now, is when you need to compile extension modules with libraries that are newer than the ones in the system, and you don't have root access. The absense of LD_LIBRARY_PATH means a segfault on an arbitrary moment during execution, not just an unloadable library.
The question is whether pythonv makes any difference here. IOW, if you used plain ol' virtualenv with some custom-location libraries, you'd have the same problem right?
If so, then in both cases you'd have to arrange for LD_LIBRARY_PATH to be set correctly for things to work. So pythonv wouldn't need to do anything special.
Cheers, -Barry
Carl Meyer wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi Mark,
On 03/18/2011 01:46 PM, Mark Sienkiewicz wrote:
Here is a case that might resemble what you are talking about:
Compile a C extension that requires a shared library that is not in the standard system path. To import it, LD_LIBRARY_PATH needs to be right.
But this required shared library that is not on the standard system path
- is it something that might have been installed, via Python
(distutils/setuptools/d2/packaging), in the virtualenv, where if it had been installed in that same way in the system Python installation it would be on the standard system path?
No. I'm thinking of examples like this:
- In some of my Macs and Solaris machines, the BLAS library does not contain all of the functions that numpy wants. I download the FORTRAN sources for BLAS, compile them, and put the libraries in $HOME/I_wish_I_had_root
- I need a newer version of the freetype library, so I ./configure --prefix $HOME/newer_freetype
python is never involved, except as an application that uses the shared libraries.
I have several different linux releases to support, so I have a lot of libraries that I install in order to have the same version on each machine.
If I had a way for python to automatically augment LD_LIBRARY_PATH when it starts up, I would use it, whether virtualenv is involved or not.
If I had a way to make distutils link shared libraries by a full path name instead of just libfoo.a, I would be happy to have that too.
In practice, I usually try to ensure that I have only static libraries available. Everything gets statically linked, so I don't need LD_LIBRARY_PATH.
If so, do you have any specific examples of installable Python projects that install a shared library that other ones might need in order to compile, so that I can test this case and make it work?
I know of an example, but I don't have much to show you that I think would be useful.
We have two packages:
pywcs does coordinate transformations on astronomical images; it is a python wrapper for wcslib (or libwcs or something), written in C.
betadrizzle does some complex manipulations of multiple images. In doing that, it sometimes needs to do huge numbers of coordinate transformations.
The betadrizzle C extension would like to call directly into the pywcs copy of wcslib, but without calling into python and back to C because of the overhead of doing it potentially millions of times.
I'm not sure what we are doing with that right now. I remember suggesting that pywcs offer some C entry points through a jump table, but I haven't caught up with the guy doing the actual work to find out what he ended up doing.
Unfortunately, betadrizzle is still an experimental project and I'm not sure I could tell you how to compile it. I am sure that I could not tell you how to run it. But you see the idea.
If not, then it seems this is a general Python packaging issue, not anything specific to this virtual-python proposal. In which case I don't have to care, at least not just now ;-)
No, you don't have to care. Shared libraries cause me way more problems than they solve, and nothing you do or don't do is going to change that. :(
Mark S.
Mark Sienkiewicz wrote:
With distutils, you can't. It goes to great lengths to ensure that you can only compile a C extension with "cc ... -L/some/directory -lname" -- I can't find any way to make it do "cc ... /some/directory/libname.so"
Are you sure you can't use extra_link_args for this?
You have rpath to record search paths for shared libraries at link time. This enables you not to set LD_LIBRARY_PATH at runtime.
Something like "cc -Wl,-rpath,/path/to/libdir -L/path/to/libdir -lfoo" is what you want.
Le 18/03/2011 18:46, Mark Sienkiewicz a écrit :
The above just means my memory is too faulty to of much use ;). I'll just echo Carl's request for specific cases where $LD_LIBRARY_PATH needs to be set.
Here is a case that might resemble what you are talking about:
Compile a C extension that requires a shared library that is not in the standard system path. To import it, LD_LIBRARY_PATH needs to be right.
This is not really different from what happens in a compiled language, except in one way: In C, I can compile it -static or I can give the full path to the .so file. Either results in a thing that works without LD_LIBRARY_PATH.
With distutils, you can't. It goes to great lengths to ensure that you can only compile a C extension with "cc ... -L/some/directory -lname" -- I can't find any way to make it do "cc ... /some/directory/libname.so"
So, the real problem here is that distutils uses "cc -L", but it demonstrates a case where LD_LIBRARY_PATH can be important to a python program even when Python itself can run without it.
Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
kiorky wrote:
You have rpath to record search paths for shared libraries at link time. This enables you not to set LD_LIBRARY_PATH at runtime.
Something like "cc -Wl,-rpath,/path/to/libdir -L/path/to/libdir -lfoo" is what you want.
You left out the hard part: How do I make distutils do that?
Le 18/03/2011 18:46, Mark Sienkiewicz a écrit :
The above just means my memory is too faulty to of much use ;). I'll just echo Carl's request for specific cases where $LD_LIBRARY_PATH needs to be set.
Here is a case that might resemble what you are talking about:
Compile a C extension that requires a shared library that is not in the standard system path. To import it, LD_LIBRARY_PATH needs to be right.
This is not really different from what happens in a compiled language, except in one way: In C, I can compile it -static or I can give the full path to the .so file. Either results in a thing that works without LD_LIBRARY_PATH.
With distutils, you can't. It goes to great lengths to ensure that you can only compile a C extension with "cc ... -L/some/directory -lname" -- I can't find any way to make it do "cc ... /some/directory/libname.so"
So, the real problem here is that distutils uses "cc -L", but it demonstrates a case where LD_LIBRARY_PATH can be important to a python program even when Python itself can run without it.
Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
export LDFLAGS="what i want to be in linker flags". You have also play the LD_RUN_PATH to modify the RPATH. See [1]
You can have a look to minitage.recipe.{common, eggs, scripts} for a good buildout integration.
[1] - http://goo.gl/oPPj0
Le 30/03/2011 23:41, Mark Sienkiewicz a écrit :
kiorky wrote:
You have rpath to record search paths for shared libraries at link time. This enables you not to set LD_LIBRARY_PATH at runtime.
Something like "cc -Wl,-rpath,/path/to/libdir -L/path/to/libdir -lfoo" is what you want.
You left out the hard part: How do I make distutils do that?
Le 18/03/2011 18:46, Mark Sienkiewicz a écrit :
The above just means my memory is too faulty to of much use ;). I'll just echo Carl's request for specific cases where $LD_LIBRARY_PATH needs to be set.
Here is a case that might resemble what you are talking about:
Compile a C extension that requires a shared library that is not in the standard system path. To import it, LD_LIBRARY_PATH needs to be right.
This is not really different from what happens in a compiled language, except in one way: In C, I can compile it -static or I can give the full path to the .so file. Either results in a thing that works without LD_LIBRARY_PATH.
With distutils, you can't. It goes to great lengths to ensure that you can only compile a C extension with "cc ... -L/some/directory -lname" -- I can't find any way to make it do "cc ... /some/directory/libname.so"
So, the real problem here is that distutils uses "cc -L", but it demonstrates a case where LD_LIBRARY_PATH can be important to a python program even when Python itself can run without it.
Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
On Wed, Mar 30, 2011 at 5:08 PM, kiorky kiorky@cryptelium.net wrote:
You have rpath to record search paths for shared libraries at link time. This enables you not to set LD_LIBRARY_PATH at runtime.
Something like "cc -Wl,-rpath,/path/to/libdir -L/path/to/libdir -lfoo" is what you want.
This fails once you want to relocate the built packages, but if you know what the installed location is going to be, can be incredibly helpful.
On 31 March 2011 13:04, Fred Drake fdrake@acm.org wrote:
On Wed, Mar 30, 2011 at 5:08 PM, kiorky kiorky@cryptelium.net wrote:
You have rpath to record search paths for shared libraries at link time. This enables you not to set LD_LIBRARY_PATH at runtime.
Something like "cc -Wl,-rpath,/path/to/libdir -L/path/to/libdir -lfoo" is what you want.
This fails once you want to relocate the built packages, but if you know what the installed location is going to be, can be incredibly helpful.
On SYSV-like systems you can use $ORIGIN inside RUNPATH (and RPATH) to create relative paths starting from the location of the library.
Regards Floris
Mark Sienkiewicz sienkiew@stsci.edu writes:
Compile a C extension that requires a shared library that is not in the standard system path. To import it, LD_LIBRARY_PATH needs to be right.
I just discovered, in the course of repairing my "pyzmq-static" project after the "pyzmq" folks went crazy and broke their single extension module into a dozen modules, that there is another alternative to the LD_LIBRARY_PATH environment variable. If an extension wants to build and use a .so shared library, then the Python package can build and install the library somewhere beneath the package's main directory, then the package can preload the library manually as the first thing __init__.py does, by writing code like this:
import ctypes import os p = os.path.join(os.path.dirname(__file__), "mylibrary.so") _library = ctypes.CDLL(p, mode=ctypes.RTLD_GLOBAL)
Because the RTLD_GLOBAL mode makes all of the symbols in "mylibrary.so" available to subsequently-loaded shared object files, any subsequent import of a dynamically linked Python extension should link its symbols against "mylibrary.so" just fine without a need to set LD_LIBRARY_PATH.
participants (15)
-
Barry Warsaw -
Brandon Craig Rhodes -
Carl Meyer -
Carl Meyer -
Floris Bruynooghe -
Fred Drake -
Greg Ewing -
Jim Fulton -
kiorky -
Leonardo Rochael Almeida -
Mark Sienkiewicz -
P.J. Eby -
Tres Seaver -
Vinay Sajip -
Éric Araujo