__pycache__ folders
Hello: I'm working on trying to get the pip test suite passing for pypy. I need to figure out the mechanism that turns on/off the use of __pycache__ folders. locally in my pypy v1.9 install, I only see normal *.pyc files for installed distributions. In Travis service builds which also reports v1.9, there are __pycache__ folders with *.pypy-19.pyc files. How is this set or determined in pypy? any help appreciated, thanks Marcus
IMHO, *.pyc files should not be distributed with a binary pypy. This is probably an issue for the team who supply the pypy for travis-ci.org (since the __pycache__ files do not appear in the nightly or release builds on pypy.org) - I think they live at https://launchpad.net/~pypy <https://launchpad.net/%7Epypy> Matti On 31/07/2012 1:28 PM, Benjamin Peterson wrote:
let me explain some more. I have may have thrown people with my use of "distribution". I just meant installed packages that you get from pypi or wherever, not the pypy distribution itself. e.g. one of our nose test cases installs and then uninstalls INITools (from pypi) it's not expecting to find a __pycache__ folder when doing an uninstall, but it's there in the travis build, despite the "installed-files.txt" manifest (in the egg-info dir) listing normal inline *.pyc files that's not so bad, except that locally, I don't know how to "turn on" __pycache__ folders for pypy so I can troubleshoot and get a workaround. I only ever get inline *.pyc files. here's the travis log for that one test. It's got some of my custom logging, so not so clear what's going on, but you can see me logging the contents of the __pycache__ dir, as the code attempts to clear the initools pkg directory. http://travis-ci.org/#!/pypa/pip/builds/1996370 thanks Marcus On Mon, Jul 30, 2012 at 8:04 PM, Marcus Smith <qwcode@gmail.com> wrote:
Hi Marcus, PyPy contains no logic to create or search for __pycache__. A grep tells me that neither does CPython 2.7. You are probably confused by py.test creating them. A bientôt, Armin.
Hi Marcus, Armin, On Tue, Jul 31, 2012 at 09:25 +0200, Armin Rigo wrote:
... creating them during a pypy test run - the artificats (*.pyc files) of such runs should not be distributed as Matti points out. Not sure it helps in this case but in general you can set PYTHONDONTWRITEBYTECODE=1 to prevent writing of such files. best, holger
Hi Armin (2012.07.31_09:25:08_+0200)
A Debian/Ubuntu pypy does, though. And IIRC, that's what travis CI is using. It's the pep3147 patches: http://anonscm.debian.org/gitweb/?p=collab-maint/pypy.git;a=tree;f=debian/pa... I thought it would be a useful feature, to make transition between pypy versions in Debian easier. Proposed the patches in #pypy, and the reception was lukewarm, so I never pushed harder. SR -- Stefano Rivera http://tumbleweed.org.za/ H: +27 21 461 1230 C: +27 72 419 8559
Hi Stefano, On Tue, Jul 31, 2012 at 11:44 AM, Stefano Rivera <stefano@rivera.za.net> wrote:
I think that PyPy shouldn't do that, because CPython doesn't. If Debian/Ubuntu also hacked CPython to do that, then fine; we are then free to point users to them (blame them?) for issues :-) A bientôt, Armin.
Hi Armin (2012.07.31_12:10:01_+0200)
We did seriously consider hacking cpython to do that, before introducing 2.7. But by that point, there was no real benefit, as 2.7 is the last of the 2.x series. https://lists.debian.org/debian-python/2010/04/msg00046.html I did it for pypy, as I thought it fairly likely that there would be a benefit (there's no guarantee that pypy's bytecode format won't change between pypy releases). Only very esoteric code would break as a result of this change. PyPy already has slightly different .pyc semantics to cpython, and has PEP3149, a vaguely similar change, for C extensions. It's worth stating that I did ask if there was any objection to the patches I was applying, more than once, before the first upload to Debian, and received none. The general opinion seemed to be that .pyc files aren't liked much in the pypy community, at all, and this was vaguely inline SR -- Stefano Rivera http://tumbleweed.org.za/ H: +27 21 461 1230 C: +27 72 419 8559
On 31.07.12 12:35, Stefano Rivera wrote:
Well, good to know. Your backport makes sense, but the reasoning should be identical between cpython and pypy. If cpython does not do it, pypy does not do it. The bytecode compatibility is also identical between cpython and pypy, there is no issue here that is not there. pypy currently shows itself as python 2.7.2, and that is the truth. Different behavior introduced by a distribution is IMHO not a benefit, but goes into an incompatible direction. If a backport happens, then in CPython, and PyPy will adjusted, accordingly. One of the incompatibility issues _is_ the unexpected behavior of tests. To Marcus: Now that the issue is clear, it probably makes sense to add some extra check to setup.py that does not inquire the python version but does some probing. Better to be defensive ;-) cheers - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Hi Christian (2012.07.31_13:05:48_+0200)
vaguely inline
Hrm, I never finished that sentance, because I went back and made the point earlier in the mail.
cpython has a long legacy in Debian, and many existing packages that depend on particular semantics. PyPy can afford to make changes that cpython can't. Using PEP3147 + 3149 means that I only need to put python modules in one place, for all versions of PyPy to consume. I don't want to end up having to carry multiple versions of pypy in the archive, as we have to with cpython 2.x. With 3.x, we've already seen the benefit of PEP3147 + 3149.
The bytecode compatibility is also identical between cpython and pypy, there is no issue here that is not there.
I believe that isn't the case: https://bitbucket.org/pypy/pypy/src/6bb09a17e2aa/pypy/module/imp/importing.p...
And I do my best to minimize that. In this case, I thought it was worth the delta. SR -- Stefano Rivera http://tumbleweed.org.za/ H: +27 21 461 1230 C: +27 72 419 8559
Hi Stefano, On Tue, Jul 31, 2012 at 1:34 PM, Stefano Rivera <stefano@rivera.za.net> wrote:
And I do my best to minimize that. In this case, I thought it was worth the delta.
My point of view is that this goes in the same pack of incompatibilities introduced by some distributions as splitting the install path (which we recommend to install as one block in /opt/pypy-1.9). It may be useful to do that according to some distribution-specific rules, but either you do it as a distribution hack and we don't care about it, or you try to come here and discuss more in details the problems that you have and how we could solve it together. As far as I know the __pycache__ is another example of distribution-specific hack that we were was not aware of. I don't think we are by principle necessarily completely opposed to it: whereas it still looks like a good idea to stick to CPython's behavior, failing that, if there are good reasons, the next best solution is probably to come with something that we can integrate into the "real" pypy, rather than keep on your own. If you don't, users complain here and you are just making us hate .pyc files even more. :-) A bientôt, Armin.
Hi Armin (2012.07.31_16:26:50_+0200)
Most distros won't use /opt. We tend to follow the FHS. /opt makes a lot of sense for the binary pypy you distribute. Just as it makes sense for the binary junk that Adobe et al. distribute.
Having the same installation layout on all distros would be fantastic. Practically, that gets non-trivial, though. Debian has the weird dist-packages layout for cpython, that I copied for pypy, but that approach hasn't been documented in upstream cpython or adopted by any other distributions.
As far as I know the __pycache__ is another example of distribution-specific hack that we were was not aware of.
I did my best to make you all aware of it, and get some feedback. I mentioned the PEP3147 support here [0], and a zillion times on IRC. But I have no problem having that conversation now that people seem to be aware of it :) [0]: http://mail.python.org/pipermail/pypy-dev/2012-January/009104.html
Yup, that's the opinion I felt before. And the approach I'm taking. But the patch is fairly gnarly due to test-suite changes, so I haven't forwarded it up to you. SR -- Stefano Rivera http://tumbleweed.org.za/ H: +27 21 461 1230 C: +27 72 419 8559
Hi, <rant> i would dare to say that Distributions that manage semi-compatible, but distro specific "forks" of open source projects instead of aiming for a distro compatible upstream create quite some grief for developers. after all users don't tend to notice those incompatibilities, but it breaks the tooling of developers I'm at a point where i need to run self-compiled pythons, since the ones in Debian for example are broken for some of my use-cases by Debian specific patches/changes what makes it worse is that the error reporting paths for the distro specific "forks" aren't made clear </rant> -- Ronny On 07/31/2012 04:26 PM, Armin Rigo wrote:
Hi Ronny (2012.07.31_17:12:15_+0200)
Weighed up against making grief for users.
after all users don't tend to notice those incompatibilities, but it breaks the tooling of developers
We really try not to make changes that break tooling. And when we do, to fix the tools, upstream.
And that was the point of the dist-packages change.
what makes it worse is that the error reporting paths for the distro specific "forks" aren't made clear
Distros have bug trackers. SR -- Stefano Rivera http://tumbleweed.org.za/ H: +27 21 461 1230 C: +27 72 419 8559
IMHO, *.pyc files should not be distributed with a binary pypy. This is probably an issue for the team who supply the pypy for travis-ci.org (since the __pycache__ files do not appear in the nightly or release builds on pypy.org) - I think they live at https://launchpad.net/~pypy <https://launchpad.net/%7Epypy> Matti On 31/07/2012 1:28 PM, Benjamin Peterson wrote:
let me explain some more. I have may have thrown people with my use of "distribution". I just meant installed packages that you get from pypi or wherever, not the pypy distribution itself. e.g. one of our nose test cases installs and then uninstalls INITools (from pypi) it's not expecting to find a __pycache__ folder when doing an uninstall, but it's there in the travis build, despite the "installed-files.txt" manifest (in the egg-info dir) listing normal inline *.pyc files that's not so bad, except that locally, I don't know how to "turn on" __pycache__ folders for pypy so I can troubleshoot and get a workaround. I only ever get inline *.pyc files. here's the travis log for that one test. It's got some of my custom logging, so not so clear what's going on, but you can see me logging the contents of the __pycache__ dir, as the code attempts to clear the initools pkg directory. http://travis-ci.org/#!/pypa/pip/builds/1996370 thanks Marcus On Mon, Jul 30, 2012 at 8:04 PM, Marcus Smith <qwcode@gmail.com> wrote:
Hi Marcus, PyPy contains no logic to create or search for __pycache__. A grep tells me that neither does CPython 2.7. You are probably confused by py.test creating them. A bientôt, Armin.
Hi Marcus, Armin, On Tue, Jul 31, 2012 at 09:25 +0200, Armin Rigo wrote:
... creating them during a pypy test run - the artificats (*.pyc files) of such runs should not be distributed as Matti points out. Not sure it helps in this case but in general you can set PYTHONDONTWRITEBYTECODE=1 to prevent writing of such files. best, holger
Hi Armin (2012.07.31_09:25:08_+0200)
A Debian/Ubuntu pypy does, though. And IIRC, that's what travis CI is using. It's the pep3147 patches: http://anonscm.debian.org/gitweb/?p=collab-maint/pypy.git;a=tree;f=debian/pa... I thought it would be a useful feature, to make transition between pypy versions in Debian easier. Proposed the patches in #pypy, and the reception was lukewarm, so I never pushed harder. SR -- Stefano Rivera http://tumbleweed.org.za/ H: +27 21 461 1230 C: +27 72 419 8559
Hi Stefano, On Tue, Jul 31, 2012 at 11:44 AM, Stefano Rivera <stefano@rivera.za.net> wrote:
I think that PyPy shouldn't do that, because CPython doesn't. If Debian/Ubuntu also hacked CPython to do that, then fine; we are then free to point users to them (blame them?) for issues :-) A bientôt, Armin.
Hi Armin (2012.07.31_12:10:01_+0200)
We did seriously consider hacking cpython to do that, before introducing 2.7. But by that point, there was no real benefit, as 2.7 is the last of the 2.x series. https://lists.debian.org/debian-python/2010/04/msg00046.html I did it for pypy, as I thought it fairly likely that there would be a benefit (there's no guarantee that pypy's bytecode format won't change between pypy releases). Only very esoteric code would break as a result of this change. PyPy already has slightly different .pyc semantics to cpython, and has PEP3149, a vaguely similar change, for C extensions. It's worth stating that I did ask if there was any objection to the patches I was applying, more than once, before the first upload to Debian, and received none. The general opinion seemed to be that .pyc files aren't liked much in the pypy community, at all, and this was vaguely inline SR -- Stefano Rivera http://tumbleweed.org.za/ H: +27 21 461 1230 C: +27 72 419 8559
On 31.07.12 12:35, Stefano Rivera wrote:
Well, good to know. Your backport makes sense, but the reasoning should be identical between cpython and pypy. If cpython does not do it, pypy does not do it. The bytecode compatibility is also identical between cpython and pypy, there is no issue here that is not there. pypy currently shows itself as python 2.7.2, and that is the truth. Different behavior introduced by a distribution is IMHO not a benefit, but goes into an incompatible direction. If a backport happens, then in CPython, and PyPy will adjusted, accordingly. One of the incompatibility issues _is_ the unexpected behavior of tests. To Marcus: Now that the issue is clear, it probably makes sense to add some extra check to setup.py that does not inquire the python version but does some probing. Better to be defensive ;-) cheers - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
Hi Christian (2012.07.31_13:05:48_+0200)
vaguely inline
Hrm, I never finished that sentance, because I went back and made the point earlier in the mail.
cpython has a long legacy in Debian, and many existing packages that depend on particular semantics. PyPy can afford to make changes that cpython can't. Using PEP3147 + 3149 means that I only need to put python modules in one place, for all versions of PyPy to consume. I don't want to end up having to carry multiple versions of pypy in the archive, as we have to with cpython 2.x. With 3.x, we've already seen the benefit of PEP3147 + 3149.
The bytecode compatibility is also identical between cpython and pypy, there is no issue here that is not there.
I believe that isn't the case: https://bitbucket.org/pypy/pypy/src/6bb09a17e2aa/pypy/module/imp/importing.p...
And I do my best to minimize that. In this case, I thought it was worth the delta. SR -- Stefano Rivera http://tumbleweed.org.za/ H: +27 21 461 1230 C: +27 72 419 8559
Hi Stefano, On Tue, Jul 31, 2012 at 1:34 PM, Stefano Rivera <stefano@rivera.za.net> wrote:
And I do my best to minimize that. In this case, I thought it was worth the delta.
My point of view is that this goes in the same pack of incompatibilities introduced by some distributions as splitting the install path (which we recommend to install as one block in /opt/pypy-1.9). It may be useful to do that according to some distribution-specific rules, but either you do it as a distribution hack and we don't care about it, or you try to come here and discuss more in details the problems that you have and how we could solve it together. As far as I know the __pycache__ is another example of distribution-specific hack that we were was not aware of. I don't think we are by principle necessarily completely opposed to it: whereas it still looks like a good idea to stick to CPython's behavior, failing that, if there are good reasons, the next best solution is probably to come with something that we can integrate into the "real" pypy, rather than keep on your own. If you don't, users complain here and you are just making us hate .pyc files even more. :-) A bientôt, Armin.
Hi Armin (2012.07.31_16:26:50_+0200)
Most distros won't use /opt. We tend to follow the FHS. /opt makes a lot of sense for the binary pypy you distribute. Just as it makes sense for the binary junk that Adobe et al. distribute.
Having the same installation layout on all distros would be fantastic. Practically, that gets non-trivial, though. Debian has the weird dist-packages layout for cpython, that I copied for pypy, but that approach hasn't been documented in upstream cpython or adopted by any other distributions.
As far as I know the __pycache__ is another example of distribution-specific hack that we were was not aware of.
I did my best to make you all aware of it, and get some feedback. I mentioned the PEP3147 support here [0], and a zillion times on IRC. But I have no problem having that conversation now that people seem to be aware of it :) [0]: http://mail.python.org/pipermail/pypy-dev/2012-January/009104.html
Yup, that's the opinion I felt before. And the approach I'm taking. But the patch is fairly gnarly due to test-suite changes, so I haven't forwarded it up to you. SR -- Stefano Rivera http://tumbleweed.org.za/ H: +27 21 461 1230 C: +27 72 419 8559
Hi, <rant> i would dare to say that Distributions that manage semi-compatible, but distro specific "forks" of open source projects instead of aiming for a distro compatible upstream create quite some grief for developers. after all users don't tend to notice those incompatibilities, but it breaks the tooling of developers I'm at a point where i need to run self-compiled pythons, since the ones in Debian for example are broken for some of my use-cases by Debian specific patches/changes what makes it worse is that the error reporting paths for the distro specific "forks" aren't made clear </rant> -- Ronny On 07/31/2012 04:26 PM, Armin Rigo wrote:
Hi Ronny (2012.07.31_17:12:15_+0200)
Weighed up against making grief for users.
after all users don't tend to notice those incompatibilities, but it breaks the tooling of developers
We really try not to make changes that break tooling. And when we do, to fix the tools, upstream.
And that was the point of the dist-packages change.
what makes it worse is that the error reporting paths for the distro specific "forks" aren't made clear
Distros have bug trackers. SR -- Stefano Rivera http://tumbleweed.org.za/ H: +27 21 461 1230 C: +27 72 419 8559
participants (9)
-
Armin Rigo -
Benjamin Peterson -
Christian Tismer -
holger krekel -
Maciej Fijalkowski -
Marcus Smith -
Matti Picus -
Ronny Pfannschmidt -
Stefano Rivera