Changing the name of __pycache__
It would be great if there was some way to change the name to .pycache so that it doesn't pollute directory listings. The dot-naming convention seems to be widely used (.bashrc, .emacs, .hgignore, etc.). Ideally, we should follow that convention also or at least provide a way to make the change locally (perhaps an environment variable). Raymond
On Tue, 8 Feb 2011 14:47:37 -0800 Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
It would be great if there was some way to change the name to .pycache so that it doesn't pollute directory listings.
The dot-naming convention seems to be widely used (.bashrc, .emacs, .hgignore, etc.). Ideally, we should follow that convention also or at least provide a way to make the change locally (perhaps an environment variable).
The fact that pyc files were not named ".foo.pyc" hints that we want them to be visible, IMO. Also, I'm not sure how a single __pycache__ directory is worse than N pyc files. Regards Antoine.
On Wed, 9 Feb 2011 09:17:46 -0800 Alexandre Conrad <alexandre.conrad@gmail.com> wrote:
Well, it does. That doesn't meant it has to be hidden itself. Besides, the main purpose is to allow for multiple cache files per source file (originally a Debian/Ubuntu request, but potentially useful for other people). That it makes directories cleaner than the old scheme is merely a side effect. Regards Antoine.
On Tue, Feb 8, 2011 at 3:15 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Right. This was quite the conscious decision when we discussed the new scheme. We want the .pyc files to be out of the way, but we DON'T want them to be completely invisible. -- --Guido van Rossum (python.org/~guido)
Raymond Hettinger wrote:
It would be great if there was some way to change the name to .pycache so that it doesn't pollute directory listings.
The dot-naming convention seems to be widely used (.bashrc, .emacs, .hgignore, etc.). Ideally, we should follow that convention also or at least provide a way to make the change locally (perhaps an environment variable).
While I don't the like name either, I think it's important that this particular aspect is not configurable: there are tools relying on finding the .pyc files based on the location of the .py files and those don't necessarily run in the same environment as the application, e.g. think of all the freeze tools, or situations where the application itself runs as daemon under a different user account than the one used to administer the application. Another use case is shipping precompiled packages. If the user changes the pyc cache dir name, the precompiled versions won't get used. BTW: I wonder how PEP 3147 will support source-less distributions. With previous versions of Python, this was easy: you just remove the .py files. With Python 3.2, removing the .py files and leaving just the files in the pyc cache will cause ImportErrors (see http://www.python.org/dev/peps/pep-3147/#id59). It seems that the only way to "build" a working source-less package is by running a special tool that moves the pyc files to where the source files lived. In addition, this mechanism does not appear to work with the new <magic> names, so distribution of packages with pycs for multiple Python versions is not possible. I'm not sure why this was done. It looks like an unnecessary limitation of the PEP. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 09 2011)
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
Raymond's suggestion is specifically addressed (and rejected) in PEP 3147: http://www.python.org/dev/peps/pep-3147/#pyc On Wed, Feb 9, 2011 at 7:07 PM, M.-A. Lemburg <mal@egenix.com> wrote:
I'm not sure why this was done. It looks like an unnecessary limitation of the PEP.
Because we wanted deleting the .py file to be the "one obvious way" to remove a module, rather than allowing "ghost modules" to hang around in __pycache__. Sourceless imports use the legacy .pyc location to distinguish them as a deliberate distribution choice over the "mere" caching in __pycache__. (Note that the original version of PEP 3147 didn't support sourceless distribution at all). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan wrote:
Isn't that a rather rare use case nowadays where packages and modules are usually installed as egg directories ? You typically delete the whole egg dir to remove an installed module/package and don't really care what's inside the directory.
Ok, but why don't those pyc files support the same <magic> add-on as the files in the __pycache__ dir ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 09 2011)
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Wed, 09 Feb 2011 12:16:28 +0100 "M.-A. Lemburg" <mal@egenix.com> wrote:
Nick was talking about deleting a single file. As a developer, not as an administrator. Bizarre issues because of a dangling pyc file after removing the py file are now the past, which is a good thing :) Regards Antoine.
On Wed, Feb 9, 2011 at 9:16 PM, M.-A. Lemburg <mal@egenix.com> wrote:
Ok, but why don't those pyc files support the same <magic> add-on as the files in the __pycache__ dir ?
Because the idea was mainly to retain the legacy .pyc support so we didn't break any sourceless distributions that already worked, not to encourage more of them. If people want to target a specific interpreter and ship sourceless, they can do that, or they can target multiple interpreter implementations by shipping the source. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan wrote:
Why alienate sourceless distributions by not supporting the same logic in the main package dir ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 09 2011)
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Thu, Feb 10, 2011 at 4:59 AM, M.-A. Lemburg <mal@egenix.com> wrote:
Why alienate sourceless distributions by not supporting the same logic in the main package dir ?
Because a large number of the people involved in the PEP 3147 discussions wanted to drop support for sourceless imports completely, including the folks doing the implementation work. Retaining legacy sourceless imports was a compromise that preserved existing functionality without increasing the implementation effort needed for the PEP. Changing that will require a patch and advocacy from folks that are actually *fans* of sourceless distribution, as opposed to merely tolerating them as an arguably necessary evil. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan wrote:
I must have missed that discussion and couldn't find it in the python-dev archives either. Otherwise I would have chimed in earlier. Do you have a pointer ?
What's evil about a sourceless distribution ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 10 2011)
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On 10/02/11 11:44, Nick Coghlan wrote:
Labelling sourceless distributions as a "necessary evil" comes across to me as a political stance, not based on any technical argument. I don't think that this kind of ideological thinking should have any place in deciding what goes into Python. -- Greg
Greg Ewing writes:
Note that Nick is *not* doing any labeling in that post (I don't know what his actual opinion is). He is saying that the people doing the work don't want to do this, and it's up to those who want it to do that work, which at this point will include "not screwing up the existing proposal." I grant that it's likely that there will be ideology-based responses of "oh, let's not do that at all", but that's not where Nick is coming from in the quoted post.
On Thu, Feb 10, 2011 at 5:04 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Yeah, I was just intending to relay the tone of the original discussion (which definitely acquired an ideological flavour at times). As I recall, I was one of those arguing that there are valid, practical use cases for sourceless distribution. However, those use cases were adequately met by the simple solution in the PEP (i.e. retaining support for substituting a bytecode file in place of a source file simply by changing the extension), so that compromise satisfied all parties involved in the discussion at the time. The best reference I found summarising the situation is here: http://www.mail-archive.com/python-dev@python.org/msg45924.html So apparently there was a fair bit of in person discussion at the PyCon 2010 language summit as well. The flurry of PyCon related list activity around that time would also explain why several developers missed the discussion. All that said, you could definitely extend the PEP 3147 idea in 3.3 to allow sourceless imports into multiple Python interpreters in a single directory, I'm just not sure I see the point. The whole goal of PEP 3147 is to allow multiple interpreters to share a single source file without fighting over a single cache location, while still keeping the cached files near the original source files. There's no such sharing benefit when it comes to sourceless distributions, so why not simply have a separate directory hierarchy per interpreter and use the basic file name? Is the desire to do this really common enough to add yet-another-stat-call to the import sequence? Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan wrote:
Thanks for the link. Now I know why I didn't spot this... I would never have assumed such a discussion under a subject line "__file__" :-)
The point is that creating sourceless distros is quite easy in Python2 and it's a feature commercial apps often want to use. They provide a way to protect your code, but are also useful to trim down the size of a distribution (the source code is not needed to use a package).
Oh yes there is :-) People often download the wrong archives for their Python version and having the possibility to ship the files for all supported Python versions in one package would make them very happy - pretty much for the same reasons the PEP makes distro maintainers happy. (And I don't really see why Linux distro maintainers are any more special than people wanting to create sourceless distributions.) Also note that this won't be another stat call everybody will have to pay for: the import logic would only fall back to the alternative pyc location in case it doesn't find the .py file, so sourceful distributions would not see any extra stat calls. You do add another stat call before raising the ImportError, but that's not really all that much of an issue. Besides, it seems noone is really worried about stat calls anymore anyway... just check how many stat calls are needed to get an interpreter started up with a few eggs sitting in site-packages. Let's add this to Python 3.3. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 10 2011)
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Thu, Feb 10, 2011 at 11:31 PM, M.-A. Lemburg <mal@egenix.com> wrote:
A fair point.
Our stat call counts are actually per-directory-on-sys.path, so even sourceful distributions see the count go up.
I forgot which list it was on (it might even have been the tracker), but there was certainly a request to look at reducing the number of stat calls for Python 3.3.
Let's add this to Python 3.3.
I certainly wouldn't object to such a change. A PEP might be advisable to hammer out details like what (if anything) to do with __file__ and the compileall command line update to request inclusion of the magic tag in the generated bytecode files. Ideally, such an interface would allow the bytecode compilation to target a separate directory to further simplify the task of generating bytecode that is separated from its original source code. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, 10 Feb 2011 15:43:06 +1300 Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
The poor debuggability of compiled-only code is certainly a technical argument (and a rather strong one). You might not like the fact that a technical argument can be used to inform political opinions, but that doesn't make that argument any less true. Regards Antoine.
On 9 February 2011 10:41, Nick Coghlan <ncoghlan@gmail.com> wrote:
Note that it looks like bdist_wininst installers don't delete the __pycache__ entries on an uninstall (which I'd argue is an error, but I'm not going to bother until I at least have time to raise an issue for it...) So satisfying imports from __pycache__ without there being a source file would break bdist_wininst uninstallation badly as things stand. Paul.
Raymond Hettinger wrote:
+1 on ".pycache". "__pycache__" looks too much like a special Python package dir to me. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 09 2011)
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Wed, 09 Feb 2011 09:07:05 -0800 Ethan Furman <ethan@stoneleaf.us> wrote:
+1 on ".pycache" as well.
Well, unless you propose postponing the forthcoming 3.2 release for that, it's probably too late anyway. (and of course it's not "just a #define"; there are tests, and probably importlib and other modules relying on it; and the PEP to update too) That said, I think it is useful that casual users of Python are aware that Python does cache bytecode files. It's not a complex enough notion that there's any point in hiding these from them. After all, explicit is better than implicit. Regards Antoine.
On Feb 9, 2011, at 9:30 AM, Antoine Pitrou wrote:
Yes, I propose that we do that now (3.2rc2). It is a simple exercise with sed to change it and not hard to get right. We've gotten +1 on .pycache from: Mark Lemburg, Ethan Furman, David Malcolm, and me. AFAICT, the only thing going for __pycache__ is that that is was is already in the tree. So far, no one has said they prefer that name to .pycache.
The dot-file naming convention is pretty well established. People use "ls" much more than they use "ls -a" because they usually don't want to see those files or directories. IOW, implicit is better when we're talking about system files and caching and whatnot. Raymond
Well, so what? Should we use any file naming convention even when it's not appropriate?
People use "ls" much more than they use "ls -a" because they usually don't want to see those files or directories.
I'm not sure people who would be annoyed by a __pycache__ directory really want to see any Python source code anyway. You only list Python directories when you are exploring or hacking something, so clearly you are interested in knowing how things work.
IOW, implicit is better when we're talking about system files and caching and whatnot.
Why so? My /var/cache directory is not named /var/.cache, and it's full of non-dotfile entries.
On Wed, Feb 9, 2011 at 12:45 PM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
FWIW, count me as "-1". I would say: "Don't fix it if it ain't broke". I've just recently started to remember where .pyc files went in recent versions and don't want to relearn the directory name again or worse learn how to figure out the name that changes from setup to setup. I'd say visibility of __pychache__ is a virtue. Users who don read "what's new" are likely to notice it and learn about the new feature.
On Wed, Feb 9, 2011 at 09:45, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
I honestly didn't think this was going to go that far, but since it is, I will say that I prefer __pycache__. I like visibly knowing that CPython has created files that it is relying upon instead of having to explicitly make sure I do `ls -a` to find out.
On UNIX. This does not extend to other platforms like Windows.
IOW, implicit is better when we're talking about system files and caching and whatnot.
I disagree. I hate the amount of dot files that accumulate in my home directory from various apps that I have used. So when I do have to go digging for some config file I suddenly discover that a bunch of apps have installed "hidden" config files that I now have to either delete or mentally ignore.
On Thu, Feb 10, 2011 at 3:45 AM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
I expect a lot folks involved in the original choice of __pycache__ as the name figured this discussion would fizzle out, and hence didn't reply, especially since PEP 3147 already made this choice explicit with Guido backing the idea for the same reasons given in this thread (http://www.python.org/dev/peps/pep-3147/#pyc) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Feb 9, 2011, at 2:38 PM, Nick Coghlan wrote:
Thanks for the link. I already dropped/retracted this idea after Guido chimed-in this morning, and I had not been aware that there was a previous discussion (I read tons of python email, docs, peps, etc but missed this particular discussion). It seems that a lot of people on this list (Guido most importantly) want to see the directory, and folks think the bytecode cache is somewhat different than .svn or .hg directories. Raymond
On 2/9/2011 12:45 PM, Raymond Hettinger wrote:
We've gotten +1 on .pycache from: Mark Lemburg, Ethan Furman, David Malcolm, and me.
A very biased sample. Knowing/remembering that the matter was already discussed and decided, I had no reason to say anything. On Windows, trying to rename a folder from 'New Folder' to '.pycache' (or anything beginning with '.') FAILS with "You must type a file name." Ditto for ordinary files. Whether the illegal name can be forced with Windows API calls that bypass the user-level check, I do not know.
AFAICT, the only thing going for __pycache__ is that that is was is already in the tree.
On Windows, it is a legal filename ;-). -- Terry Jan Reedy
On Thu, Feb 10, 2011 at 5:31 PM, Terry Reedy <tjreedy@udel.edu> wrote:
You can create dot-files from the Windows command prompt. Windows Explorer just has additional prejudices as to what constitutes a valid filename. Dot-files are definitely a point of pain when dealing with Unix-ish programs on Windows, though. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 02/09/2011 10:02 AM, Raymond Hettinger wrote:
-1 I personally don't like hidden directories and files of any type. I think it is very good that python avoids those where it can. Hidden files and directories have their own problems. They can be forgotten or missed. If there is a problem associated with a hidden file, it can lead to a lot of wasted time when people look for problems else where because they are not readily aware the hidden files or directories. Please don't use hidden files or directories, the use of a single directory is the best balance of keeping things out of the way, yet not hiding them totally. Ron
Raymond Hettinger wrote:
If a consensus were to emerge, it would still be possible to change the name from "__pycache__" to ".pycache".
-1. Please don't encourage the Unix anti-pattern of scattering invisible breadcrumbs all throughout your work-area. Besides, unless I'm misinformed, such dot files aren't invisible in Windows systems (or Mac?), so the fundamental assumption that changing the name will make it invisible will be wrong for many, perhaps most, users. I don't particularly like the name __pycache__ but it does match the Python convention of using double-underscore names. Otherwise, it risks clashing with a user's own directory. I *far* prefer it over .pycache. -- Steven
2011/2/9 Steven D'Aprano <steve@pearwood.info>:
Yes, I did think about that afterwards.
"dunder" naming is a Python convention and is OK for Python code. Even though I am not a big fan of the __init__.py file, at least the user created it for his python package to be seen. Whereas __pycache__ conveys the idea that it is has a special meaning (such as __init__.py) and suggests it may alter your application's behavior, which does not. Just nitpicking, I guess. So I remove my +1 for the ".pycache" idea (for the reason that being hidden is platform specific) and my call is now +0. -- Alex | twitter.com/alexconrad
Steven D'Aprano <steve@pearwood.info> writes:
+1 from me.
The anti-pattern is the scattering of breadcrumbs. I agree with discouraging that practice. But Python already breaks that, and PEP 3147 is an attempt at making the practice less messy. The use of leading-dot names to make them invisible is a good feature. If breadcrumbs must be scattered, at least keep them out of the way in normal use.
Then ‘.pycache’ is no harm on such systems, surely? -- \ “Those are my principles. If you don't like them I have | `\ others.” —Groucho Marx | _o__) | Ben Finney
On Tue, 8 Feb 2011 14:47:37 -0800 Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
It would be great if there was some way to change the name to .pycache so that it doesn't pollute directory listings.
The dot-naming convention seems to be widely used (.bashrc, .emacs, .hgignore, etc.). Ideally, we should follow that convention also or at least provide a way to make the change locally (perhaps an environment variable).
The fact that pyc files were not named ".foo.pyc" hints that we want them to be visible, IMO. Also, I'm not sure how a single __pycache__ directory is worse than N pyc files. Regards Antoine.
On Wed, 9 Feb 2011 09:17:46 -0800 Alexandre Conrad <alexandre.conrad@gmail.com> wrote:
Well, it does. That doesn't meant it has to be hidden itself. Besides, the main purpose is to allow for multiple cache files per source file (originally a Debian/Ubuntu request, but potentially useful for other people). That it makes directories cleaner than the old scheme is merely a side effect. Regards Antoine.
On Tue, Feb 8, 2011 at 3:15 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
Right. This was quite the conscious decision when we discussed the new scheme. We want the .pyc files to be out of the way, but we DON'T want them to be completely invisible. -- --Guido van Rossum (python.org/~guido)
Raymond Hettinger wrote:
It would be great if there was some way to change the name to .pycache so that it doesn't pollute directory listings.
The dot-naming convention seems to be widely used (.bashrc, .emacs, .hgignore, etc.). Ideally, we should follow that convention also or at least provide a way to make the change locally (perhaps an environment variable).
While I don't the like name either, I think it's important that this particular aspect is not configurable: there are tools relying on finding the .pyc files based on the location of the .py files and those don't necessarily run in the same environment as the application, e.g. think of all the freeze tools, or situations where the application itself runs as daemon under a different user account than the one used to administer the application. Another use case is shipping precompiled packages. If the user changes the pyc cache dir name, the precompiled versions won't get used. BTW: I wonder how PEP 3147 will support source-less distributions. With previous versions of Python, this was easy: you just remove the .py files. With Python 3.2, removing the .py files and leaving just the files in the pyc cache will cause ImportErrors (see http://www.python.org/dev/peps/pep-3147/#id59). It seems that the only way to "build" a working source-less package is by running a special tool that moves the pyc files to where the source files lived. In addition, this mechanism does not appear to work with the new <magic> names, so distribution of packages with pycs for multiple Python versions is not possible. I'm not sure why this was done. It looks like an unnecessary limitation of the PEP. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 09 2011)
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
Raymond's suggestion is specifically addressed (and rejected) in PEP 3147: http://www.python.org/dev/peps/pep-3147/#pyc On Wed, Feb 9, 2011 at 7:07 PM, M.-A. Lemburg <mal@egenix.com> wrote:
I'm not sure why this was done. It looks like an unnecessary limitation of the PEP.
Because we wanted deleting the .py file to be the "one obvious way" to remove a module, rather than allowing "ghost modules" to hang around in __pycache__. Sourceless imports use the legacy .pyc location to distinguish them as a deliberate distribution choice over the "mere" caching in __pycache__. (Note that the original version of PEP 3147 didn't support sourceless distribution at all). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan wrote:
Isn't that a rather rare use case nowadays where packages and modules are usually installed as egg directories ? You typically delete the whole egg dir to remove an installed module/package and don't really care what's inside the directory.
Ok, but why don't those pyc files support the same <magic> add-on as the files in the __pycache__ dir ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 09 2011)
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Wed, 09 Feb 2011 12:16:28 +0100 "M.-A. Lemburg" <mal@egenix.com> wrote:
Nick was talking about deleting a single file. As a developer, not as an administrator. Bizarre issues because of a dangling pyc file after removing the py file are now the past, which is a good thing :) Regards Antoine.
On Wed, Feb 9, 2011 at 9:16 PM, M.-A. Lemburg <mal@egenix.com> wrote:
Ok, but why don't those pyc files support the same <magic> add-on as the files in the __pycache__ dir ?
Because the idea was mainly to retain the legacy .pyc support so we didn't break any sourceless distributions that already worked, not to encourage more of them. If people want to target a specific interpreter and ship sourceless, they can do that, or they can target multiple interpreter implementations by shipping the source. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan wrote:
Why alienate sourceless distributions by not supporting the same logic in the main package dir ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 09 2011)
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Thu, Feb 10, 2011 at 4:59 AM, M.-A. Lemburg <mal@egenix.com> wrote:
Why alienate sourceless distributions by not supporting the same logic in the main package dir ?
Because a large number of the people involved in the PEP 3147 discussions wanted to drop support for sourceless imports completely, including the folks doing the implementation work. Retaining legacy sourceless imports was a compromise that preserved existing functionality without increasing the implementation effort needed for the PEP. Changing that will require a patch and advocacy from folks that are actually *fans* of sourceless distribution, as opposed to merely tolerating them as an arguably necessary evil. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan wrote:
I must have missed that discussion and couldn't find it in the python-dev archives either. Otherwise I would have chimed in earlier. Do you have a pointer ?
What's evil about a sourceless distribution ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 10 2011)
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On 10/02/11 11:44, Nick Coghlan wrote:
Labelling sourceless distributions as a "necessary evil" comes across to me as a political stance, not based on any technical argument. I don't think that this kind of ideological thinking should have any place in deciding what goes into Python. -- Greg
Greg Ewing writes:
Note that Nick is *not* doing any labeling in that post (I don't know what his actual opinion is). He is saying that the people doing the work don't want to do this, and it's up to those who want it to do that work, which at this point will include "not screwing up the existing proposal." I grant that it's likely that there will be ideology-based responses of "oh, let's not do that at all", but that's not where Nick is coming from in the quoted post.
On Thu, Feb 10, 2011 at 5:04 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Yeah, I was just intending to relay the tone of the original discussion (which definitely acquired an ideological flavour at times). As I recall, I was one of those arguing that there are valid, practical use cases for sourceless distribution. However, those use cases were adequately met by the simple solution in the PEP (i.e. retaining support for substituting a bytecode file in place of a source file simply by changing the extension), so that compromise satisfied all parties involved in the discussion at the time. The best reference I found summarising the situation is here: http://www.mail-archive.com/python-dev@python.org/msg45924.html So apparently there was a fair bit of in person discussion at the PyCon 2010 language summit as well. The flurry of PyCon related list activity around that time would also explain why several developers missed the discussion. All that said, you could definitely extend the PEP 3147 idea in 3.3 to allow sourceless imports into multiple Python interpreters in a single directory, I'm just not sure I see the point. The whole goal of PEP 3147 is to allow multiple interpreters to share a single source file without fighting over a single cache location, while still keeping the cached files near the original source files. There's no such sharing benefit when it comes to sourceless distributions, so why not simply have a separate directory hierarchy per interpreter and use the basic file name? Is the desire to do this really common enough to add yet-another-stat-call to the import sequence? Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan wrote:
Thanks for the link. Now I know why I didn't spot this... I would never have assumed such a discussion under a subject line "__file__" :-)
The point is that creating sourceless distros is quite easy in Python2 and it's a feature commercial apps often want to use. They provide a way to protect your code, but are also useful to trim down the size of a distribution (the source code is not needed to use a package).
Oh yes there is :-) People often download the wrong archives for their Python version and having the possibility to ship the files for all supported Python versions in one package would make them very happy - pretty much for the same reasons the PEP makes distro maintainers happy. (And I don't really see why Linux distro maintainers are any more special than people wanting to create sourceless distributions.) Also note that this won't be another stat call everybody will have to pay for: the import logic would only fall back to the alternative pyc location in case it doesn't find the .py file, so sourceful distributions would not see any extra stat calls. You do add another stat call before raising the ImportError, but that's not really all that much of an issue. Besides, it seems noone is really worried about stat calls anymore anyway... just check how many stat calls are needed to get an interpreter started up with a few eggs sitting in site-packages. Let's add this to Python 3.3. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 10 2011)
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Thu, Feb 10, 2011 at 11:31 PM, M.-A. Lemburg <mal@egenix.com> wrote:
A fair point.
Our stat call counts are actually per-directory-on-sys.path, so even sourceful distributions see the count go up.
I forgot which list it was on (it might even have been the tracker), but there was certainly a request to look at reducing the number of stat calls for Python 3.3.
Let's add this to Python 3.3.
I certainly wouldn't object to such a change. A PEP might be advisable to hammer out details like what (if anything) to do with __file__ and the compileall command line update to request inclusion of the magic tag in the generated bytecode files. Ideally, such an interface would allow the bytecode compilation to target a separate directory to further simplify the task of generating bytecode that is separated from its original source code. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, 10 Feb 2011 15:43:06 +1300 Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
The poor debuggability of compiled-only code is certainly a technical argument (and a rather strong one). You might not like the fact that a technical argument can be used to inform political opinions, but that doesn't make that argument any less true. Regards Antoine.
On 9 February 2011 10:41, Nick Coghlan <ncoghlan@gmail.com> wrote:
Note that it looks like bdist_wininst installers don't delete the __pycache__ entries on an uninstall (which I'd argue is an error, but I'm not going to bother until I at least have time to raise an issue for it...) So satisfying imports from __pycache__ without there being a source file would break bdist_wininst uninstallation badly as things stand. Paul.
Raymond Hettinger wrote:
+1 on ".pycache". "__pycache__" looks too much like a special Python package dir to me. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 09 2011)
::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
On Wed, 09 Feb 2011 09:07:05 -0800 Ethan Furman <ethan@stoneleaf.us> wrote:
+1 on ".pycache" as well.
Well, unless you propose postponing the forthcoming 3.2 release for that, it's probably too late anyway. (and of course it's not "just a #define"; there are tests, and probably importlib and other modules relying on it; and the PEP to update too) That said, I think it is useful that casual users of Python are aware that Python does cache bytecode files. It's not a complex enough notion that there's any point in hiding these from them. After all, explicit is better than implicit. Regards Antoine.
On Feb 9, 2011, at 9:30 AM, Antoine Pitrou wrote:
Yes, I propose that we do that now (3.2rc2). It is a simple exercise with sed to change it and not hard to get right. We've gotten +1 on .pycache from: Mark Lemburg, Ethan Furman, David Malcolm, and me. AFAICT, the only thing going for __pycache__ is that that is was is already in the tree. So far, no one has said they prefer that name to .pycache.
The dot-file naming convention is pretty well established. People use "ls" much more than they use "ls -a" because they usually don't want to see those files or directories. IOW, implicit is better when we're talking about system files and caching and whatnot. Raymond
Well, so what? Should we use any file naming convention even when it's not appropriate?
People use "ls" much more than they use "ls -a" because they usually don't want to see those files or directories.
I'm not sure people who would be annoyed by a __pycache__ directory really want to see any Python source code anyway. You only list Python directories when you are exploring or hacking something, so clearly you are interested in knowing how things work.
IOW, implicit is better when we're talking about system files and caching and whatnot.
Why so? My /var/cache directory is not named /var/.cache, and it's full of non-dotfile entries.
On Wed, Feb 9, 2011 at 12:45 PM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
FWIW, count me as "-1". I would say: "Don't fix it if it ain't broke". I've just recently started to remember where .pyc files went in recent versions and don't want to relearn the directory name again or worse learn how to figure out the name that changes from setup to setup. I'd say visibility of __pychache__ is a virtue. Users who don read "what's new" are likely to notice it and learn about the new feature.
On Wed, Feb 9, 2011 at 09:45, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
I honestly didn't think this was going to go that far, but since it is, I will say that I prefer __pycache__. I like visibly knowing that CPython has created files that it is relying upon instead of having to explicitly make sure I do `ls -a` to find out.
On UNIX. This does not extend to other platforms like Windows.
IOW, implicit is better when we're talking about system files and caching and whatnot.
I disagree. I hate the amount of dot files that accumulate in my home directory from various apps that I have used. So when I do have to go digging for some config file I suddenly discover that a bunch of apps have installed "hidden" config files that I now have to either delete or mentally ignore.
On Thu, Feb 10, 2011 at 3:45 AM, Raymond Hettinger <raymond.hettinger@gmail.com> wrote:
I expect a lot folks involved in the original choice of __pycache__ as the name figured this discussion would fizzle out, and hence didn't reply, especially since PEP 3147 already made this choice explicit with Guido backing the idea for the same reasons given in this thread (http://www.python.org/dev/peps/pep-3147/#pyc) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Feb 9, 2011, at 2:38 PM, Nick Coghlan wrote:
Thanks for the link. I already dropped/retracted this idea after Guido chimed-in this morning, and I had not been aware that there was a previous discussion (I read tons of python email, docs, peps, etc but missed this particular discussion). It seems that a lot of people on this list (Guido most importantly) want to see the directory, and folks think the bytecode cache is somewhat different than .svn or .hg directories. Raymond
On 2/9/2011 12:45 PM, Raymond Hettinger wrote:
We've gotten +1 on .pycache from: Mark Lemburg, Ethan Furman, David Malcolm, and me.
A very biased sample. Knowing/remembering that the matter was already discussed and decided, I had no reason to say anything. On Windows, trying to rename a folder from 'New Folder' to '.pycache' (or anything beginning with '.') FAILS with "You must type a file name." Ditto for ordinary files. Whether the illegal name can be forced with Windows API calls that bypass the user-level check, I do not know.
AFAICT, the only thing going for __pycache__ is that that is was is already in the tree.
On Windows, it is a legal filename ;-). -- Terry Jan Reedy
On Thu, Feb 10, 2011 at 5:31 PM, Terry Reedy <tjreedy@udel.edu> wrote:
You can create dot-files from the Windows command prompt. Windows Explorer just has additional prejudices as to what constitutes a valid filename. Dot-files are definitely a point of pain when dealing with Unix-ish programs on Windows, though. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 02/09/2011 10:02 AM, Raymond Hettinger wrote:
-1 I personally don't like hidden directories and files of any type. I think it is very good that python avoids those where it can. Hidden files and directories have their own problems. They can be forgotten or missed. If there is a problem associated with a hidden file, it can lead to a lot of wasted time when people look for problems else where because they are not readily aware the hidden files or directories. Please don't use hidden files or directories, the use of a single directory is the best balance of keeping things out of the way, yet not hiding them totally. Ron
Raymond Hettinger wrote:
If a consensus were to emerge, it would still be possible to change the name from "__pycache__" to ".pycache".
-1. Please don't encourage the Unix anti-pattern of scattering invisible breadcrumbs all throughout your work-area. Besides, unless I'm misinformed, such dot files aren't invisible in Windows systems (or Mac?), so the fundamental assumption that changing the name will make it invisible will be wrong for many, perhaps most, users. I don't particularly like the name __pycache__ but it does match the Python convention of using double-underscore names. Otherwise, it risks clashing with a user's own directory. I *far* prefer it over .pycache. -- Steven
2011/2/9 Steven D'Aprano <steve@pearwood.info>:
Yes, I did think about that afterwards.
"dunder" naming is a Python convention and is OK for Python code. Even though I am not a big fan of the __init__.py file, at least the user created it for his python package to be seen. Whereas __pycache__ conveys the idea that it is has a special meaning (such as __init__.py) and suggests it may alter your application's behavior, which does not. Just nitpicking, I guess. So I remove my +1 for the ".pycache" idea (for the reason that being hidden is platform specific) and my call is now +0. -- Alex | twitter.com/alexconrad
Steven D'Aprano <steve@pearwood.info> writes:
+1 from me.
The anti-pattern is the scattering of breadcrumbs. I agree with discouraging that practice. But Python already breaks that, and PEP 3147 is an attempt at making the practice less messy. The use of leading-dot names to make them invisible is a good feature. If breadcrumbs must be scattered, at least keep them out of the way in normal use.
Then ‘.pycache’ is no harm on such systems, surely? -- \ “Those are my principles. If you don't like them I have | `\ others.” —Groucho Marx | _o__) | Ben Finney
participants (16)
-
Alexander Belopolsky -
Alexandre Conrad -
Antoine Pitrou -
Ben Finney -
Brett Cannon -
Ethan Furman -
Greg Ewing -
Guido van Rossum -
M.-A. Lemburg -
Nick Coghlan -
Paul Moore -
Raymond Hettinger -
Ron Adam -
Stephen J. Turnbull -
Steven D'Aprano -
Terry Reedy