Re: [Python-Dev] PEP 3147: PYC Repository Directories
On Feb 03, 2010, at 11:07 PM, Nick Coghlan wrote:
It's also the case that having to run Python to manage my own filesystem would very annoying. If a dev has a broken .pyc that prevents the affected Python build from even starting how are they meant to use the nonfunctioning interpreter to find and delete the offending file? How is someone meant to find and delete the .pyc files if they prefer to use a graphical file manager over (or in conjunction with) the command line?
I agree. I'd prefer to have a predictable place for the cached files, independent of having to run Python to tell you where that is. -Barry
Barry Warsaw <barry@python.org> writes:
On Feb 03, 2010, at 11:07 PM, Nick Coghlan wrote:
It's also the case that having to run Python to manage my own filesystem would very annoying. […]
Files that are problematic wouldn't need Python to manage any more than currently. The suggestion was just that, a suggestion for Python to expose information to assist; it wouldn't be required.
I agree. I'd prefer to have a predictable place for the cached files, independent of having to run Python to tell you where that is.
Right; I don't see who would disagree with that. I don't see any conflict between “decouple compiled bytecode file locations from source file locations” versus “predictable location for the compiled bytecode files”. -- \ “All television is educational television. The question is: | `\ what is it teaching?” —Nicholas Johnson | _o__) | Ben Finney
On Sat, Feb 6, 2010 at 4:27 PM, Ben Finney <ben+python@benfinney.id.au> wrote:
Barry Warsaw <barry@python.org> writes:
I agree. I'd prefer to have a predictable place for the cached files, independent of having to run Python to tell you where that is.
Right; I don't see who would disagree with that. I don't see any conflict between “decouple compiled bytecode file locations from source file locations” versus “predictable location for the compiled bytecode files”.
The conflict is purely that PEP 3147 proposes the new behavior to be optional, and adds a flag (-R) and an environment variable ($PYTHONPYR) to change it. I presume Barry is proposing this out of fear that the new behavior might upset somebody; personally I think it would be better if the behavior weren't optional. At least not in new Python releases -- in backports such as a distribution that wants this feature might make, it may make sense to be more conservative, or at least to have a way to turn it off. -- --Guido van Rossum (python.org/~guido)
On Feb 06, 2010, at 04:39 PM, Guido van Rossum wrote:
The conflict is purely that PEP 3147 proposes the new behavior to be optional, and adds a flag (-R) and an environment variable ($PYTHONPYR) to change it. I presume Barry is proposing this out of fear that the new behavior might upset somebody; personally I think it would be better if the behavior weren't optional. At least not in new Python releases
Good to know! Yes, that's one reason why I made it option, the other being that I suspect most people don't care about the original use case (making sure pyc files from different Python versions don't conflict). However, with a folder-per-folder approach, the side benefit of reducing directory clutter by hiding all the pyc files becomes more compelling.
-- in backports such as a distribution that wants this feature might make, it may make sense to be more conservative, or at least to have a way to turn it off.
For backports I think the most conservative approach is to require a flag to enable this behavior. If we make this the default for new versions of Python (something I'd support) then tools written for Python >= 3.2 will know this is just how it's done. I worry about existing deployed tools for Python < 2.7 and 3.1. How about this: enable it by default in 3.2 and 2.7. No option to disable it. Allow distro back ports to define a flag or environment variable to enable it. The PEP can even be silent about how that's actually done, and a Debian implementation for Python 2.6 or 3.1 could even use the (now documented :) -X flag. -Barry
On Sun, Feb 7, 2010 at 10:44, Barry Warsaw <barry@python.org> wrote:
On Feb 06, 2010, at 04:39 PM, Guido van Rossum wrote:
The conflict is purely that PEP 3147 proposes the new behavior to be optional, and adds a flag (-R) and an environment variable ($PYTHONPYR) to change it. I presume Barry is proposing this out of fear that the new behavior might upset somebody; personally I think it would be better if the behavior weren't optional. At least not in new Python releases
Good to know! Yes, that's one reason why I made it option, the other being that I suspect most people don't care about the original use case (making sure pyc files from different Python versions don't conflict). However, with a folder-per-folder approach, the side benefit of reducing directory clutter by hiding all the pyc files becomes more compelling.
-- in backports such as a distribution that wants this feature might make, it may make sense to be more conservative, or at least to have a way to turn it off.
For backports I think the most conservative approach is to require a flag to enable this behavior. If we make this the default for new versions of Python (something I'd support) then tools written for Python >= 3.2 will know this is just how it's done. I worry about existing deployed tools for Python < 2.7 and 3.1.
How about this: enable it by default in 3.2 and 2.7. No option to disable it. Allow distro back ports to define a flag or environment variable to enable it. The PEP can even be silent about how that's actually done, and a Debian implementation for Python 2.6 or 3.1 could even use the (now documented :) -X flag.
Would you keep the old behavior around as well, or simply drop it? I personally vote for the latter for simplicity and performance reasons (by not having to look in so many places for bytecode), but I can see tool people who magically calculate the location of the bytecode not loving the idea (another reason why giving loaders a method to return all relevant paths is a good idea; no more guessing). -Brett
-Barry
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org
On Sun, Feb 7, 2010 at 12:23 PM, Brett Cannon <brett@python.org> wrote:
On Sun, Feb 7, 2010 at 10:44, Barry Warsaw <barry@python.org> wrote:
On Feb 06, 2010, at 04:39 PM, Guido van Rossum wrote:
The conflict is purely that PEP 3147 proposes the new behavior to be optional, and adds a flag (-R) and an environment variable ($PYTHONPYR) to change it. I presume Barry is proposing this out of fear that the new behavior might upset somebody; personally I think it would be better if the behavior weren't optional. At least not in new Python releases
Good to know! Yes, that's one reason why I made it option, the other being that I suspect most people don't care about the original use case (making sure pyc files from different Python versions don't conflict). However, with a folder-per-folder approach, the side benefit of reducing directory clutter by hiding all the pyc files becomes more compelling.
-- in backports such as a distribution that wants this feature might make, it may make sense to be more conservative, or at least to have a way to turn it off.
For backports I think the most conservative approach is to require a flag to enable this behavior. If we make this the default for new versions of Python (something I'd support) then tools written for Python >= 3.2 will know this is just how it's done. I worry about existing deployed tools for Python < 2.7 and 3.1.
How about this: enable it by default in 3.2 and 2.7. No option to disable it. Allow distro back ports to define a flag or environment variable to enable it. The PEP can even be silent about how that's actually done, and a Debian implementation for Python 2.6 or 3.1 could even use the (now documented :) -X flag.
Would you keep the old behavior around as well, or simply drop it? I personally vote for the latter for simplicity and performance reasons (by not having to look in so many places for bytecode), but I can see tool people who magically calculate the location of the bytecode not loving the idea (another reason why giving loaders a method to return all relevant paths is a good idea; no more guessing).
For 3.2 I think it's fine to simply drop the old behavior (as long as a good loader API is added at the same time). But for 2.7 I think we ought to be a lot more conservative and not force tools to upgrade, so I think we should keep the old behavior in 2.7 as the default (though distros can change this if they want to, and backport if they need to). -- --Guido van Rossum (python.org/~guido)
Ben Finney wrote:
Right; I don't see who would disagree with that. I don't see any conflict between “decouple compiled bytecode file locations from source file locations” versus “predictable location for the compiled bytecode files”.
The more decoupled they are, the harder it is to manually find the bytecode file. With the current .pyc scheme, .pyr folders or an SVN style Python cache directory, finding the bytecode file is pretty easy, since the cached file is either in the same directory as the source file or in a subdirectory. With any form of shadow hierarchy though, it gets trickier because you have to: 1. Find the root of the shadow hierarchy 2. Navigate within the shadow hierarchy down to the point that matches where your source file was It's a fairly significant increase in mental overhead. It gets much worse if the location of the shadow hierarchy root is configurable in any way (e.g. based on sys.path contents or an environment variable). Restricting the caching mechanism to the folder containing the source file keeps things a lot simpler. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Sat, Feb 6, 2010 at 5:10 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Ben Finney wrote:
Right; I don't see who would disagree with that. I don't see any conflict between “decouple compiled bytecode file locations from source file locations” versus “predictable location for the compiled bytecode files”.
The more decoupled they are, the harder it is to manually find the bytecode file.
With the current .pyc scheme, .pyr folders or an SVN style Python cache directory, finding the bytecode file is pretty easy, since the cached file is either in the same directory as the source file or in a subdirectory.
With any form of shadow hierarchy though, it gets trickier because you have to: 1. Find the root of the shadow hierarchy 2. Navigate within the shadow hierarchy down to the point that matches where your source file was
It's a fairly significant increase in mental overhead. It gets much worse if the location of the shadow hierarchy root is configurable in any way (e.g. based on sys.path contents or an environment variable).
Restricting the caching mechanism to the folder containing the source file keeps things a lot simpler.
Great way of explaining why the basic folder-per-folder model wins over the folder-per-sys.path-entry model! The basic folder-per-folder model doesn't need to know what sys.path is. (And I hadn't followed previous messages in the thread with enough care to understand the subtlen implications of Ben's point. Sorry!) -- --Guido van Rossum (python.org/~guido)
Nick Coghlan <ncoghlan@gmail.com> writes:
The more decoupled they are, the harder it is to manually find the bytecode file.
Okay. So it's not so much about “predictable”, but rather about “predictable by a human without too much cognitive effort”. I can see value in that, though it's best to be explicit that this is a goal (to be clear that “a program can tell you where they live” isn't a solution).
It's a fairly significant increase in mental overhead. It gets much worse if the location of the shadow hierarchy root is configurable in any way (e.g. based on sys.path contents or an environment variable).
Restricting the caching mechanism to the folder containing the source file keeps things a lot simpler.
Simpler for the human working on the source code; not for the human trying to fit this scheme in with an OS package management system. (Again, I'm just clarifying and making the contrast explicit, not judging relative values.) This makes it clearer to me that there is a glaring incompatibility between this desire for “keep the compiled bytecode files close to the source files” versus “decouple the locations so the OS package manager can do its job of managing installed files”. I recognise after earlier discussion in this thread that's not an issue being addressed by PEP 3147. -- \ “Those are my principles. If you don't like them I have | `\ others.” —Groucho Marx | _o__) | Ben Finney
participants (5)
-
Barry Warsaw -
Ben Finney -
Brett Cannon -
Guido van Rossum -
Nick Coghlan