I understand the need to ship without source -- but why does that require supporting .pyc (or .pyo) -only? Couldn't vendors just replace the real .py files with empty files? Then no one would need the extra stat call, and no one would be bitten by orphaned .pyc files after a rename. [Yes, zips could still allow unmatched names; yes, it would be helpful if a tool were available to sync the last-modification time; yes a deprecation release should still be needed.] -jJ
On Mar 03, 2010, at 07:37 PM, Jim Jewett wrote:
I understand the need to ship without source -- but why does that require supporting .pyc (or .pyo) -only?
Couldn't vendors just replace the real .py files with empty files?
Yes, I think that's a possibility. What would people think about that? -Barry
On approximately 3/3/2010 5:49 PM, came the following characters from the keyboard of Barry Warsaw:
On Mar 03, 2010, at 07:37 PM, Jim Jewett wrote:
I understand the need to ship without source -- but why does that require supporting .pyc (or .pyo) -only?
Couldn't vendors just replace the real .py files with empty files?
Yes, I think that's a possibility. What would people think about that?
-Barry
That's kooky, but not as kooky as my idea. As mentioned elsewhere, timestamps would have to be treated carefully. In this scenario, the .pyc files would still live in __pycache__ ? Complete with the foo.<token>.pyc naming ? So one could actually create a "fat .zip" application that could work with a variety of installed Python versions ??? Hmm... -- Glenn -- http://nevcal.com/ =========================== A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking
On Mar 3, 2010, at 10:22 PM, Greg Ewing wrote:
Glenn Linderman wrote:
In this scenario, the .pyc files would still live in __pycache__ ? Complete with the foo.<token>.pyc naming ?
It might be neater to have a separate cache directory for each bytecode version, named __cache.<token>__ or some such.
Okay, this is probably some pretty silly bikeshedding, but: if we're going to have it be something.something-else, can we please make sure that .something-else is a common extension that means "python bytecode cache"? It would be good to keep the file-manager and shell operations required to say "blow away bytecode cache directories" as simple as possible.
Le Thu, 04 Mar 2010 16:22:13 +1300, Greg Ewing <greg.ewing@canterbury.ac.nz> a écrit :
Glenn Linderman wrote:
In this scenario, the .pyc files would still live in __pycache__ ? Complete with the foo.<token>.pyc naming ?
It might be neater to have a separate cache directory for each bytecode version, named __cache.<token>__ or some such.
Actually, I find it neater to have a single cache directory. It makes for much less clutter, and simpler ignore rules.
Antoine Pitrou wrote:
Actually, I find it neater to have a single cache directory. It makes for much less clutter, and simpler ignore rules.
Another possibility would be to have a single top-level cache directory with a subdirectory for each version: __bytecode__.pycache/<version>/<filename>.pyc I don't think this ought to increase the stat call count, since the interpreter will always be looking for a particular version of bytecode, so it can just stat for the appropriate subdirectory directly. -- Greg
Barry Warsaw wrote:
On Mar 03, 2010, at 07:37 PM, Jim Jewett wrote:
Couldn't vendors just replace the real .py files with empty files?
Yes, I think that's a possibility. What would people think about that?
Seems like a perverse thing to have to do to me. Also a bit fragile, since you would have to make sure that the empty .py files were dated older than the .pyc files and stayed that way, lest Python try to recompile them and wipe out your code. You would also have to be careful to build installers that didn't recompile .py files on installation. (Haven't had much experience building installers using distutils, so I'm not sure how much of a problem that would be.) -- Greg
Barry Warsaw wrote:
On Mar 03, 2010, at 07:37 PM, Jim Jewett wrote:
I understand the need to ship without source -- but why does that require supporting .pyc (or .pyo) -only?
Couldn't vendors just replace the real .py files with empty files?
Yes, I think that's a possibility. What would people think about that?
I actually thought of this, but didn't post it because it defeats the point of byte-code only distribution. - putting text in the .py file will break the application - touching the .py file in any way will break the application If someone wants to break the bytecode only support it should be its own PEP and not coupled with PEP 3147. The remaining open question to my mind is whether or not there should be a -X option to control the bytecode generation. E.g.: -Xcache_bytecode=no (don't write bytecode files at all) -Xcache_bytecode=file (write a classic "foo.pyc" file) -Xcache_bytecode=dir (write to the "__pycache__" directory) With cache_bytecode=dir being the default for future releases. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Mar 04, 2010, at 11:34 PM, Nick Coghlan wrote:
The remaining open question to my mind is whether or not there should be a -X option to control the bytecode generation. E.g.:
-Xcache_bytecode=no (don't write bytecode files at all)
-B and $PYTHONDONTWRITEBYTECODE will still be supported and serve this use case.
-Xcache_bytecode=file (write a classic "foo.pyc" file) -Xcache_bytecode=dir (write to the "__pycache__" directory)
With cache_bytecode=dir being the default for future releases.
I believe we've decided /not/ to support creation of bytecode-only distributions out of the box. -Barry
On 12/03/2010 19:53, Barry Warsaw wrote:
On Mar 04, 2010, at 11:34 PM, Nick Coghlan wrote:
The remaining open question to my mind is whether or not there should be a -X option to control the bytecode generation. E.g.:
-Xcache_bytecode=no (don't write bytecode files at all)
-B and $PYTHONDONTWRITEBYTECODE will still be supported and serve this use case.
-Xcache_bytecode=file (write a classic "foo.pyc" file) -Xcache_bytecode=dir (write to the "__pycache__" directory)
With cache_bytecode=dir being the default for future releases.
I believe we've decided /not/ to support creation of bytecode-only distributions out of the box.
I thought Guido said on this topic [1]: " FWIW, I started at -1 and am still -1. I think the PEP is overreaching in this aspect; it does not serve the stated purpose of the PEP to make life easier for distros that share code between Python versions." Has something changed since then? Michael [1] http://mail.python.org/pipermail/python-dev/2010-March/098098.html
-Barry
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
On 12/03/2010 22:48, Michael Foord wrote:
On 12/03/2010 19:53, Barry Warsaw wrote:
On Mar 04, 2010, at 11:34 PM, Nick Coghlan wrote:
The remaining open question to my mind is whether or not there should be a -X option to control the bytecode generation. E.g.:
-Xcache_bytecode=no (don't write bytecode files at all)
-B and $PYTHONDONTWRITEBYTECODE will still be supported and serve this use case.
-Xcache_bytecode=file (write a classic "foo.pyc" file) -Xcache_bytecode=dir (write to the "__pycache__" directory)
With cache_bytecode=dir being the default for future releases.
I believe we've decided /not/ to support creation of bytecode-only distributions out of the box.
I thought Guido said on this topic [1]:
Ok - so oops, I misunderstood. I'll go back into my corner now. :-) All the best, Michael
" FWIW, I started at -1 and am still -1. I think the PEP is overreaching in this aspect; it does not serve the stated purpose of the PEP to make life easier for distros that share code between Python versions."
Has something changed since then?
Michael
[1] http://mail.python.org/pipermail/python-dev/2010-March/098098.html
-Barry
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe:http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog
READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.u...
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
On Mar 12, 2010, at 10:48 PM, Michael Foord wrote:
On 12/03/2010 19:53, Barry Warsaw wrote:
I believe we've decided /not/ to support creation of bytecode-only distributions out of the box.
I thought Guido said on this topic [1]:
" FWIW, I started at -1 and am still -1. I think the PEP is overreaching in this aspect; it does not serve the stated purpose of the PEP to make life easier for distros that share code between Python versions."
Has something changed since then?
Nope, sorry I should have been clearer. *creation* is the key here. As per BDFL pronouncement, we'll support reading pyc-only modules just like we do today. This is in PEP 3147. We won't support creating them though. BTW, I'm actually starting to work on the implementation now, so if you want to play along: % bzr branch lp:~barry/python/pep3147 -Barry
Barry Warsaw wrote:
*creation* is the key here. As per BDFL pronouncement, we'll support reading pyc-only modules just like we do today. This is in PEP 3147. We won't support creating them though.
The creation side could be made a little more explicit in the PEP. We could also do something via the compileall module. (Pause while Nick goes and reads the source code for compileall for the first time ever...) Hmm - methinks the PEP actually needs to talk explicitly about the py_compile and compileall modules. These compile the files directly rather than using the import system's side-effect, so they'll need to understand the intricacies of the new system. While it's probably OK if the import side-effects only create files using the new scheme, the standard library modules will likely need to support both schemes (although I'm not sure if "same as import system" or "same as Python 3.1" make more sense as the default semantics - probably the former). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On 13 March 2010 14:17, Nick Coghlan <ncoghlan@gmail.com> wrote:
The creation side could be made a little more explicit in the PEP. We could also do something via the compileall module.
(Pause while Nick goes and reads the source code for compileall for the first time ever...)
Hmm - methinks the PEP actually needs to talk explicitly about the py_compile and compileall modules. These compile the files directly rather than using the import system's side-effect, so they'll need to understand the intricacies of the new system.
The bdist_wininst installer also compiles modules explicitly on install (as does the python.org Windows MSI installer). I've always assumed that this worked via compileall, but haven't checked. Regardless, these should probably also be covered in the PEP. Paul.
On Mar 14, 2010, at 04:37 PM, Paul Moore wrote:
The bdist_wininst installer also compiles modules explicitly on install (as does the python.org Windows MSI installer). I've always assumed that this worked via compileall, but haven't checked. Regardless, these should probably also be covered in the PEP.
Thanks; added to PEP. -Barry
On Mar 14, 2010, at 12:17 AM, Nick Coghlan wrote:
Hmm - methinks the PEP actually needs to talk explicitly about the py_compile and compileall modules. These compile the files directly rather than using the import system's side-effect, so they'll need to understand the intricacies of the new system.
Good point. I'll add this to the PEP.
While it's probably OK if the import side-effects only create files using the new scheme, the standard library modules will likely need to support both schemes (although I'm not sure if "same as import system" or "same as Python 3.1" make more sense as the default semantics - probably the former).
I don't understand this point. compileall probably /could/ be extended to understand bytecode-only (i.e. legacy or <3.2) layout. I've added that to the PEP too. -Barry
Barry Warsaw wrote:
On Mar 14, 2010, at 12:17 AM, Nick Coghlan wrote:
While it's probably OK if the import side-effects only create files using the new scheme, the standard library modules will likely need to support both schemes (although I'm not sure if "same as import system" or "same as Python 3.1" make more sense as the default semantics - probably the former).
I don't understand this point.
compileall probably /could/ be extended to understand bytecode-only (i.e. legacy or <3.2) layout. I've added that to the PEP too.
In Python 3.1, *invoking* py_compile.compile() will create 2.x style bytecode. Similarly, when force==False, compileall.compile_dir() and compileall.compile_path() will check for 2.x style bytecode in order to decide whether or not to compile the module. The question for 3.2 is what bytecode layout py_compile.compile() should generate. For the precompile-a-system-library use case it should clearly generate a PEP 3147 layout and this probably makes sense as the default behaviour in 3.2. However, for production of bytecode-only packages, it would be convenient to be able to explicitly invoke the 2.x style behaviour without having to specify the target filename explicitly using the 'cfile' parameter (which isn't exposed at the compileall layer anyway). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Mar 16, 2010, at 07:44 PM, Nick Coghlan wrote:
In Python 3.1, *invoking* py_compile.compile() will create 2.x style bytecode. Similarly, when force==False, compileall.compile_dir() and compileall.compile_path() will check for 2.x style bytecode in order to decide whether or not to compile the module.
The question for 3.2 is what bytecode layout py_compile.compile() should generate. For the precompile-a-system-library use case it should clearly generate a PEP 3147 layout and this probably makes sense as the default behaviour in 3.2.
However, for production of bytecode-only packages, it would be convenient to be able to explicitly invoke the 2.x style behaviour without having to specify the target filename explicitly using the 'cfile' parameter (which isn't exposed at the compileall layer anyway).
My working branch modifies py_compile to produce PEP 3147 layout by default. I agree that it should support traditional pyc output as an option, and compileall should support this as well. -Barry
On Fri, Mar 12, 2010 at 05:56:57PM -0500, Barry Warsaw wrote:
Nope, sorry I should have been clearer.
*creation* is the key here. As per BDFL pronouncement, we'll support reading pyc-only modules just like we do today. This is in PEP 3147. We won't support creating them though.
BTW, I'm actually starting to work on the implementation now, so if you want to play along:
% bzr branch lp:~barry/python/pep3147
Sorry if I missed it, but why on earth is the bytecode directory __pycache__ and not .pycache? (Or indeed anything else that starts with a '.') Surely this is a classic ideal use case for a "hidden" directory?
On Sun, 14 Mar 2010 11:59:50 pm Jon Ribbens wrote:
Sorry if I missed it, but why on earth is the bytecode directory __pycache__ and not .pycache? (Or indeed anything else that starts with a '.') Surely this is a classic ideal use case for a "hidden" directory?
I disagree with your assumption that there is *any* use-case for a hidden directory, let alone an ideal one. I despise hidden directories and dot files. I know it is the "Unix way", and I suppose it made sense back in 1975 when users had two or three dot files in their home directory, but I count 215 dot files in my home directory compared to only 77 visible files, and I have no idea how most of them got there or what they do. Programs that litter the file system with dot files are bad enough when they do it in $HOME, but sprinkling dot files everywhere they can is inexcusable. This is not the place for me to rant over the evil that is dot files, so I'll just say this: Python works on platforms other than Unix/Linux, and some of those platforms don't treat dot files as anything more than a file with a leading dot in the file name. -- Steven D'Aprano
Le Sun, 14 Mar 2010 12:59:50 +0000, Jon Ribbens <jon+python-dev@unequivocal.co.uk> a écrit :
Sorry if I missed it, but why on earth is the bytecode directory __pycache__ and not .pycache? (Or indeed anything else that starts with a '.') Surely this is a classic ideal use case for a "hidden" directory?
I suppose the same reason pyc files are not hidden nowadays. We want people to be aware that these files exist.
On Sun, 14 Mar 2010 12:59:50 -0000, Jon Ribbens <jon+python-dev@unequivocal.co.uk> wrote:
On Fri, Mar 12, 2010 at 05:56:57PM -0500, Barry Warsaw wrote:
Nope, sorry I should have been clearer.
*creation* is the key here. As per BDFL pronouncement, we'll support reading pyc-only modules just like we do today. This is in PEP 3147. We won't support creating them though.
BTW, I'm actually starting to work on the implementation now, so if you want to play along:
% bzr branch lp:~barry/python/pep3147
Sorry if I missed it, but why on earth is the bytecode directory __pycache__ and not .pycache? (Or indeed anything else that starts with a '.') Surely this is a classic ideal use case for a "hidden" directory?
You did miss it. Perhaps a link to the discussion should be added to the PEP, or a summary of it. If I remember correctly the BDFL even weighed in on this point. -- R. David Murray www.bitdance.com
R. David Murray wrote:
On Sun, 14 Mar 2010 12:59:50 -0000, Jon Ribbens <jon+python-dev@unequivocal.co.uk> wrote:
Sorry if I missed it, but why on earth is the bytecode directory __pycache__ and not .pycache? (Or indeed anything else that starts with a '.') Surely this is a classic ideal use case for a "hidden" directory?
You did miss it. Perhaps a link to the discussion should be added to the PEP, or a summary of it. If I remember correctly the BDFL even weighed in on this point.
He did (in favour of keeping the directory visible). Here's one reference (although not the main one): http://osdir.com/ml/python-dev/2010-02/msg00114.html My Google-fu is failing me, so I don't have a link to the other part of the thread that I refer to in that message. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Mar 15, 2010, at 07:43 AM, Nick Coghlan wrote:
He did (in favour of keeping the directory visible).
http://www.mail-archive.com/python-dev@python.org/msg45203.html (added to pep) -Barry
Jon Ribbens wrote:
Sorry if I missed it, but why on earth is the bytecode directory __pycache__ and not .pycache? (Or indeed anything else that starts with a '.') Surely this is a classic ideal use case for a "hidden" directory?
Please don't try to hide it. On MacOSX, it makes it impossible to look inside using the Finder if you want to, and it won't have any effect on Windows anyway. -- Greg
On Wed, Mar 3, 2010 at 16:37, Jim Jewett <jimjjewett@gmail.com> wrote:
I understand the need to ship without source -- but why does that require supporting .pyc (or .pyo) -only?
Couldn't vendors just replace the real .py files with empty files?
Because if someone screws up the mod time on the source files the .pyc files will get recreated silently. -Brett
Then no one would need the extra stat call, and no one would be bitten by orphaned .pyc files after a rename.
[Yes, zips could still allow unmatched names; yes, it would be helpful if a tool were available to sync the last-modification time; yes a deprecation release should still be needed.]
-jJ _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org
Le Wed, 3 Mar 2010 17:51:04 -0800, Brett Cannon <brett@python.org> a écrit :
On Wed, Mar 3, 2010 at 16:37, Jim Jewett <jimjjewett@gmail.com> wrote:
I understand the need to ship without source -- but why does that require supporting .pyc (or .pyo) -only?
Couldn't vendors just replace the real .py files with empty files?
Because if someone screws up the mod time on the source files the .pyc files will get recreated silently.
Unless the .py files arrange to raise a syntax error on compiling.
Antoine Pitrou wrote:
Unless the .py files arrange to raise a syntax error on compiling.
I guess that prevents a total disaster, but the program is still broken and you have to hunt down the offending files and fix the timestamps -- if it's even evident what the problem is and how to fix it. -- Greg
Le Thu, 04 Mar 2010 16:39:03 +1300, Greg Ewing <greg.ewing@canterbury.ac.nz> a écrit :
Antoine Pitrou wrote:
Unless the .py files arrange to raise a syntax error on compiling.
I guess that prevents a total disaster, but the program is still broken and you have to hunt down the offending files and fix the timestamps -- if it's even evident what the problem is and how to fix it.
Bytecode-only distributions are supposed to be created by the vendor/distributor. It's up to him/her to get things right. Regards Antoine.
participants (13)
-
Antoine Pitrou -
Barry Warsaw -
Brett Cannon -
Glenn Linderman -
Glyph Lefkowitz -
Greg Ewing -
Jim Jewett -
Jon Ribbens -
Michael Foord -
Nick Coghlan -
Paul Moore -
R. David Murray -
Steven D'Aprano