PYTHONUSERBASES (plural!)
I was excited to see PYTHONUSERBASE added in 2.6/3.0. But I became disappointed when I learned it only allows you to specify one directory. I have use cases where I'd want more than one. Also, PYTHONUSERBASE doesn't support all the directories that site.addsitepackages does. PYTHONUSERBASE examines exactly one; on non-Windows platforms, it's: {prefix}/lib/pythonX.X/site-packages However, for each "prefix" directory Python knows about, site.addsitepackages examines these two on Linux: {prefix}/lib/pythonX.X/site-packages {prefix}/lib/site-python (The list is even longer on OS X.) I therefore propose a new environment variable called PYTHONUSERBASES. PYTHONUSERBASES is a list of directories separated by whatever directory separator characters PYTHONPATH permits on the local platform. At startup, after processing PYTHONUSERBASE but before calling site.addsitepackages, Python iterates down the list of directories listed in PYTHONUSERBASES and causes them to be processed by site.addsitepackages before the default prefix directories. (Probably by adding them to sys.prefixes.) PYTHONUSERBASES obeys -s, PYTHONNOUSERSITE, and the rules about effective user/group IDs, just as PYTHONUSERBASE does. The default value of PYTHONUSERBASES is the empty string; this would add no new directories. It seems to me that PYTHONUSERBASES would turn "virtualenv" into a twenty-line shell script. Add the appropriate directories to LD_LIBRARY_PATH and PYTHONUSERBASES and you'd be off to the races. You'd have to be careful to manually specify --prefix when installing new software, which I'm guessing virtualenv obviates. But apart from that I think it would work fine. Or am I missing something important? I would be happy--thrilled!, even--to write this up as a patch for Python 3.1 before the feature-freeze, /larry/
Larry Hastings <larry@hastings.org> writes:
I like the described feature. Adding a new variable whose meaning entirely obsoletes the existing one with confusingly similar names, especially since the existing one is itself so new, seems sub-optimal. Instead of adding another variable, I would prefer the meaning of the existing ‘PYTHONUSERBASE’ altered to do this. -- \ “I know you believe you understood what you think I said, but I | `\ am not sure you realize that what you heard is not what I | _o__) meant.” —Robert J. McCloskey | Ben Finney
Ben Finney wrote:
I like the described feature.
You vote +1 then?
I don't think we should break PYTHONUSERBASE. Also, PYTHONUSERBASES has slightly different semantics; PYTHONUSERBASE has a default value (~/.local on UNIX for instance), wheras my proposed variable has a no-op default. I'm open to suggestions for an alternate name. However, the flipside to "confusingly similar" would be "confusingly different". PYTHONUSERBASES is very similar to PYTHONUSERBASE except that it supports more than one, so giving it a different name seemed natural to me. /larry/
Larry Hastings <larry@hastings.org> writes:
With the caveat that I think it should be ‘PYTHONUSERBASE’ that causes this behaviour, not a new variable.
As I understand it, a valid value for ‘PYTHONUSERBASE’ as-is (i.e. a single directory path) is also a valid value for this new behaviour. All we're doing is adding the capability to *also* supply multiple directory paths in the value, separated by ‘:’. It seems logical to me that this should be the same variable. -- \ “Some mornings, it's just not worth chewing through the leather | `\ straps.” —Emo Philips | _o__) | Ben Finney
Scott David Daniels wrote:
So when I switch between two virtualized environments, you suggest that I remove the .pth files from the old one and add the .pth files from the new one? This strikes you as a good solution? If I have two virtualized environments, and I want to work with them concurrently, and they each require different versions of the same library, how do I achieve that with the current mechanism? /larry/
Larry Hastings wrote:
You would have two different directories each with a different .pth file in it and point PYTHONUSERBASE at the correct one for the virtual environment you want to use. PYTHONUSERBASE would still be the switching point - including a .pth file just lets you extend a given value for PYTHONUSERBASE to mean more than one directory. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Nick Coghlan wrote:
This would preclude finding packages in my default PYTHONUSERBASE. Or combining any two virtualized environments together. I suppose that's solvable if we go back to copying files around, like a cheap "virtualenv". I could make a temporary directory, copy in the lowest-priority packages and .pth files, then copy in the next-highest, and so on for all "user bases" I want to compose together. Then point PYTHONUSERBASE at that. But I'd really rather have something like PYTHONUSERBASES. For the record, would you be for or against PYTHONUSERBASES? I think I'm going to knock together a patch tonight; I hope you'll support it. /larry// / p.s. Having ruminated about it some more, I'm going to change the name to PYTHONPREFIXPATH. Like "PYTHONPATH", but for "prefix" directories (as in "site.PREFIXES"). Still open to suggestions though.//
Larry Hastings <larry@hastings.org> writes:
But I'd really rather have something like PYTHONUSERBASES. For the record, would you be for or against PYTHONUSERBASES?
I'm for the feature as an extension of what we have, and against the implementation of a new variable for that feature. It belongs as an extended interpretation of PYTHONUSERBASE, to my understanding. -- \ “Holy polar ice sheet, Batman!” —Robin | `\ | _o__) | Ben Finney
Larry Hastings wrote:
Put "my_default_dir.pth" in each of your virtual environments (preferably via a symlink if you aren't on Windows). If you want to mix two environments together, put both of their .pth files in a single directory (again via symlink if you can). A collection of .pth files in a directory is easy to manage. Messing with environment variables is a PITA - you need an external solution to keep track of what the environment variable should contain for each environment. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Nick Coghlan wrote:
But to do as you suggest, you also need to "mess with environment variables": PYTHONUSERBASE. Unless you propose changing ~/.local every time you switch environments, you need to change that variable every time you switch environments. Therefore your proposed solution must modify exactly same amount of environment variables as mine. Your proposal also adds overhead maintaining those collections of .pth files. For example, if you add a new package to one of your virtualized environments, you'd need to update every .pth collection directory which references that environment. Even ignoring those details I still don't agree. I think adding to a single environment variable is far cleaner than copying/linking files around on disks. It can be done easily on a command-by-command basis, or stowed in a shell script or an alias, not to mention easily managed by a virtualized environment system like virtualenv. Shall I put you down as a -1? (I'm not having a lot of luck getting numeric votes outta folks.) /larry/
Larry Hastings <larry@hastings.org> writes:
Shall I put you down as a -1? (I'm not having a lot of luck getting numeric votes outta folks.)
Perhaps that should be interpreted as a preference for discussing ideas and finding the good *and* bad in them, rather than artificially distilling the discussion to a linear vote. -- \ “Our products just aren't engineered for security.” —Brian | `\ Valentine, senior vice-president of Microsoft Windows | _o__) development, 2002 | Ben Finney
Ben Finney wrote:
Surely numeric voting in Python newsgroups has a long and rich tradition. And I don't think "-1" is a misrepresentation of Nick's reaction to my proposal. But I do await his reply. While we're on the subject, I've just posted a patch, here: http://bugs.python.org/issue5819 Note that I had a change of heart and renamed the environment variable, as someone suggested. The current name is PYTHONPREFIXES. I'm going to nip off and post about it to python-dev, /larry/
Larry Hastings <larry@hastings.org> writes:
Yes, and with good reason: it's a useful short cut *if* the person merely wants to say they unilaterally support or oppose a proposal. I'm suggesting that, on a forum dedicated to *discussing ideas* which are hopefully interesting enough to be worth thinking about and discussing, you're perhaps oversimplifying if you hope the responses to an idea can be usefully distilled to one of those two.
While we're on the subject, I've just posted a patch, here:
Thanks for proceeding with this work without waiting for votes :-) -- \ “[W]hoever is able to make you absurd is able to make you | `\ unjust.” —Voltaire | _o__) | Ben Finney
Larry Hastings wrote:
If you're monkeying with an environment variable, then why not just modify PYTHONPATH directly? Also, messing with environment variables on Windows is a real PITA (largely because there is no env equivalent).
Shall I put you down as a -1? (I'm not having a lot of luck getting numeric votes outta folks.)
Yep, it currently gets a "-1, redundant" from me. However, I think you would get much less push back if you: 1. Articulated clearly the specific scenarios you want to support beyond the simple single user site-packages equivalent that PYTHONUSERBASE was designed to support 2. Described why PYTHONUSERBASE + .pth files don't support them well 3. Described why traditional sys.path altering techniques (e.g. via PYTHONPATH) or tools like virtualenv aren't adequate to address these scenarios 4. Described precisely how your new mechanism improves support for the identified scenarios. So far it appears to me that you've picked up the shiny new PYTHONUSERBASE hammer and are trying to hit a screw with it when there are plenty of existing screwdrivers lying around. However, I don't know if that is actually what is going on, or if I just haven't understood the use cases you are wanting to support. Cheers, Nick. P.S. I'm moving house tomorrow and don't know when I will get my net connection back. So don't worry too much about trying to persuade me - I probably won't be around until after the first 3.1 beta has already happened. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Larry Hastings <larry@hastings.org> writes:
I like the described feature. Adding a new variable whose meaning entirely obsoletes the existing one with confusingly similar names, especially since the existing one is itself so new, seems sub-optimal. Instead of adding another variable, I would prefer the meaning of the existing ‘PYTHONUSERBASE’ altered to do this. -- \ “I know you believe you understood what you think I said, but I | `\ am not sure you realize that what you heard is not what I | _o__) meant.” —Robert J. McCloskey | Ben Finney
Ben Finney wrote:
I like the described feature.
You vote +1 then?
I don't think we should break PYTHONUSERBASE. Also, PYTHONUSERBASES has slightly different semantics; PYTHONUSERBASE has a default value (~/.local on UNIX for instance), wheras my proposed variable has a no-op default. I'm open to suggestions for an alternate name. However, the flipside to "confusingly similar" would be "confusingly different". PYTHONUSERBASES is very similar to PYTHONUSERBASE except that it supports more than one, so giving it a different name seemed natural to me. /larry/
Larry Hastings <larry@hastings.org> writes:
With the caveat that I think it should be ‘PYTHONUSERBASE’ that causes this behaviour, not a new variable.
As I understand it, a valid value for ‘PYTHONUSERBASE’ as-is (i.e. a single directory path) is also a valid value for this new behaviour. All we're doing is adding the capability to *also* supply multiple directory paths in the value, separated by ‘:’. It seems logical to me that this should be the same variable. -- \ “Some mornings, it's just not worth chewing through the leather | `\ straps.” —Emo Philips | _o__) | Ben Finney
Scott David Daniels wrote:
So when I switch between two virtualized environments, you suggest that I remove the .pth files from the old one and add the .pth files from the new one? This strikes you as a good solution? If I have two virtualized environments, and I want to work with them concurrently, and they each require different versions of the same library, how do I achieve that with the current mechanism? /larry/
Larry Hastings wrote:
You would have two different directories each with a different .pth file in it and point PYTHONUSERBASE at the correct one for the virtual environment you want to use. PYTHONUSERBASE would still be the switching point - including a .pth file just lets you extend a given value for PYTHONUSERBASE to mean more than one directory. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Nick Coghlan wrote:
This would preclude finding packages in my default PYTHONUSERBASE. Or combining any two virtualized environments together. I suppose that's solvable if we go back to copying files around, like a cheap "virtualenv". I could make a temporary directory, copy in the lowest-priority packages and .pth files, then copy in the next-highest, and so on for all "user bases" I want to compose together. Then point PYTHONUSERBASE at that. But I'd really rather have something like PYTHONUSERBASES. For the record, would you be for or against PYTHONUSERBASES? I think I'm going to knock together a patch tonight; I hope you'll support it. /larry// / p.s. Having ruminated about it some more, I'm going to change the name to PYTHONPREFIXPATH. Like "PYTHONPATH", but for "prefix" directories (as in "site.PREFIXES"). Still open to suggestions though.//
Larry Hastings <larry@hastings.org> writes:
But I'd really rather have something like PYTHONUSERBASES. For the record, would you be for or against PYTHONUSERBASES?
I'm for the feature as an extension of what we have, and against the implementation of a new variable for that feature. It belongs as an extended interpretation of PYTHONUSERBASE, to my understanding. -- \ “Holy polar ice sheet, Batman!” —Robin | `\ | _o__) | Ben Finney
Larry Hastings wrote:
Put "my_default_dir.pth" in each of your virtual environments (preferably via a symlink if you aren't on Windows). If you want to mix two environments together, put both of their .pth files in a single directory (again via symlink if you can). A collection of .pth files in a directory is easy to manage. Messing with environment variables is a PITA - you need an external solution to keep track of what the environment variable should contain for each environment. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
Nick Coghlan wrote:
But to do as you suggest, you also need to "mess with environment variables": PYTHONUSERBASE. Unless you propose changing ~/.local every time you switch environments, you need to change that variable every time you switch environments. Therefore your proposed solution must modify exactly same amount of environment variables as mine. Your proposal also adds overhead maintaining those collections of .pth files. For example, if you add a new package to one of your virtualized environments, you'd need to update every .pth collection directory which references that environment. Even ignoring those details I still don't agree. I think adding to a single environment variable is far cleaner than copying/linking files around on disks. It can be done easily on a command-by-command basis, or stowed in a shell script or an alias, not to mention easily managed by a virtualized environment system like virtualenv. Shall I put you down as a -1? (I'm not having a lot of luck getting numeric votes outta folks.) /larry/
Larry Hastings <larry@hastings.org> writes:
Shall I put you down as a -1? (I'm not having a lot of luck getting numeric votes outta folks.)
Perhaps that should be interpreted as a preference for discussing ideas and finding the good *and* bad in them, rather than artificially distilling the discussion to a linear vote. -- \ “Our products just aren't engineered for security.” —Brian | `\ Valentine, senior vice-president of Microsoft Windows | _o__) development, 2002 | Ben Finney
Ben Finney wrote:
Surely numeric voting in Python newsgroups has a long and rich tradition. And I don't think "-1" is a misrepresentation of Nick's reaction to my proposal. But I do await his reply. While we're on the subject, I've just posted a patch, here: http://bugs.python.org/issue5819 Note that I had a change of heart and renamed the environment variable, as someone suggested. The current name is PYTHONPREFIXES. I'm going to nip off and post about it to python-dev, /larry/
Larry Hastings <larry@hastings.org> writes:
Yes, and with good reason: it's a useful short cut *if* the person merely wants to say they unilaterally support or oppose a proposal. I'm suggesting that, on a forum dedicated to *discussing ideas* which are hopefully interesting enough to be worth thinking about and discussing, you're perhaps oversimplifying if you hope the responses to an idea can be usefully distilled to one of those two.
While we're on the subject, I've just posted a patch, here:
Thanks for proceeding with this work without waiting for votes :-) -- \ “[W]hoever is able to make you absurd is able to make you | `\ unjust.” —Voltaire | _o__) | Ben Finney
Larry Hastings wrote:
If you're monkeying with an environment variable, then why not just modify PYTHONPATH directly? Also, messing with environment variables on Windows is a real PITA (largely because there is no env equivalent).
Shall I put you down as a -1? (I'm not having a lot of luck getting numeric votes outta folks.)
Yep, it currently gets a "-1, redundant" from me. However, I think you would get much less push back if you: 1. Articulated clearly the specific scenarios you want to support beyond the simple single user site-packages equivalent that PYTHONUSERBASE was designed to support 2. Described why PYTHONUSERBASE + .pth files don't support them well 3. Described why traditional sys.path altering techniques (e.g. via PYTHONPATH) or tools like virtualenv aren't adequate to address these scenarios 4. Described precisely how your new mechanism improves support for the identified scenarios. So far it appears to me that you've picked up the shiny new PYTHONUSERBASE hammer and are trying to hit a screw with it when there are plenty of existing screwdrivers lying around. However, I don't know if that is actually what is going on, or if I just haven't understood the use cases you are wanting to support. Cheers, Nick. P.S. I'm moving house tomorrow and don't know when I will get my net connection back. So don't worry too much about trying to persuade me - I probably won't be around until after the first 3.1 beta has already happened. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
participants (4)
-
Ben Finney -
Larry Hastings -
Nick Coghlan -
Scott David Daniels