Mailman 3 February 2010 - Python-Dev

Re: [Python-Dev] PEP 3147: PYC Repository Directories
by Barry Warsaw Feb. 9, 2010

Feb. 9, 2010

On Jan 31, 2010, at 01:06 PM, Ron Adam wrote: >With a single cache directory, we could have an option to force writing >bytecode to a desired location. That might be useful on it's own for >creating runtime bytecode only installations for installers. One important reason for wanting to keep the bytecode cache files colocated with the source files is that I want to be able to continue to manipulate $PYTHONPATH to control how Python finds its modules. With a single system-wide cache directory that won't be easy. E.g. $PYTHONPATH might be hacked to find the source file you expect, but how would that interact with how Python finds its cache files? I'm strongly in favor of keeping the cache files as close to the source they were generated from as possible. -Barry

4 4

Re: [Python-Dev] PEP 3146: Merge Unladen Swallow into CPython
by Collin Winter Feb. 9, 2010

Feb. 9, 2010

Hi Craig, On Tue, Feb 2, 2010 at 4:42 PM, Craig Citro <craigcitro(a)gmail.com> wrote: >> Done. The diff is at >> http://codereview.appspot.com/186247/diff2/5014:8003/7002. I listed >> Cython, Shedskin and a bunch of other alternatives to pure CPython. >> Some of that information is based on conversations I've had with the >> respective developers, and I'd appreciate corrections if I'm out of >> date. >> > > Well, it's a minor nit, but it might be more fair to say something > like "Cython provides the biggest improvements once type annotations > are added to the code." After all, Cython is more than happy to take > arbitrary Python code as input -- it's just much more effective when > it knows something about types. The code to make Cython handle > closures has just been merged ... hopefully support for the full > Python language isn't so far off. (Let me know if you want me to > actually make a comment on Rietveld ...) Indeed, you're quite right. I've corrected the description here: http://codereview.appspot.com/186247/diff2/7005:9001/10001 > Now what's more interesting is whether or not U-S and Cython could > play off one another -- take a Python program, run it with some > "generic input data" under Unladen and record info about which > functions are hot, and what types they tend to take, then let > Cython/gcc -O3 have a go at these, and lather, rinse, repeat ... JIT > compilation and static compilation obviously serve different purposes, > but I'm curious if there aren't other interesting ways to take > advantage of both. Definitely! Someone approached me about possibly reusing the profile data for a feedback-enhanced code coverage tool, which has interesting potential, too. I've added a note about this under the "Future Work" section: http://codereview.appspot.com/186247/diff2/9001:10002/9003 Thanks, Collin Winter

1 0

__file__ is not always an absolute path
by Ezio Melotti Feb. 8, 2010

Feb. 8, 2010

In #7712 I was trying to change regrtest to always run the tests in a temporary CWD (e.g. /tmp/@test_1234_cwd/). The patches attached to the issue add a context manager that changes the CWD, and it works fine when I run ./python -m test.regrtest from trunk/. However, when I try from trunk/Lib/ it fails with ImportErrors (note that the latest patch by Florent Xicluna already tries to workaround the problem). The traceback points to "the_package = __import__(abstest, globals(), locals(), [])" in runtest_inner (in regrtest.py), and a "print __import__('test').__file__" there returns 'test/__init__.pyc'. This can be reproduced quite easily: trunk$ ./python Python 2.7a2+ (trunk:77941M, Feb 3 2010, 06:40:49) [GCC 4.4.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import os, sys >>> os.getcwd() '/home/wolf/dev/trunk' >>> import test >>> test.__file__ # absolute '/home/wolf/dev/trunk/Lib/test/__init__.pyc' >>> os.chdir('/tmp') >>> test.__file__ '/home/wolf/dev/trunk/Lib/test/__init__.pyc' >>> from test import test_unicode # works >>> test_unicode.__file__ '/home/wolf/dev/trunk/Lib/test/test_unicode.pyc' >>> [21]+ Stopped ./python trunk$ cd Lib/ trunk/Lib$ ../python Python 2.7a2+ (trunk:77941M, Feb 3 2010, 06:40:49) [GCC 4.4.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import os, sys >>> os.getcwd() '/home/wolf/dev/trunk/Lib' >>> import test >>> test.__file__ # relative 'test/__init__.pyc' >>> os.chdir('/tmp') >>> from test import test_unicode # fails Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: cannot import name test_unicode Is there a reason why in the second case test.__file__ is relative?

8 13

Re: [Python-Dev] PEP 3147: PYC Repository Directories
by Barry Warsaw Feb. 8, 2010

Feb. 8, 2010

On Feb 03, 2010, at 11:07 PM, Nick Coghlan wrote: >It's also the case that having to run Python to manage my own filesystem >would very annoying. If a dev has a broken .pyc that prevents the >affected Python build from even starting how are they meant to use the >nonfunctioning interpreter to find and delete the offending file? How is >someone meant to find and delete the .pyc files if they prefer to use a >graphical file manager over (or in conjunction with) the command line? I agree. I'd prefer to have a predictable place for the cached files, independent of having to run Python to tell you where that is. -Barry

5 8

Re: [Python-Dev] PEP 3147: PYC Repository Directories
by Barry Warsaw Feb. 7, 2010

Feb. 7, 2010

On Jan 31, 2010, at 11:34 PM, Nick Coghlan wrote: >I must admit I quite like the __pyr__ directory approach as well. Since >the interpreter knows the suffix it is looking for, names shouldn't >conflict. Using a single directory allows the name to be less cryptic, >too (e.g. __pycache__). Something else that occurs to me; the name of the directory (under folder-per-folder approach) probably ought to be the same as the name of the module attribute. There's probably no good reason to make it different, and making it the same makes the association stronger. That still gives us plenty of opportunity to bikeshed, but __pycache__ seems reasonable to me (it's the cache of parsing and compiling the .py file). -Barry

2 1

Re: [Python-Dev] PEP 3147: PYC Repository Directories
by Barry Warsaw Feb. 7, 2010

Feb. 7, 2010

On Feb 03, 2010, at 11:59 AM, M.-A. Lemburg wrote: >How about using an optionally relative cache dir setting to let >the user decide ? Why do we need that level of flexibility? -Barry

2 1

Re: [Python-Dev] PEP 3147: PYC Repository Directories
by Barry Warsaw Feb. 7, 2010

Feb. 7, 2010

On Feb 03, 2010, at 01:17 PM, Guido van Rossum wrote: >Can you clarify? In Python 3, __file__ always points to the source. >Clearly that is the way of the future. For 99.99% of uses of __file__, >if it suddenly never pointed to a .pyc file any more (even if one >existed) that would be just fine. So what's this talk of switching to >__source__? Upon further reflection, I agree. __file__ also points to the source in Python 2.7. Do we need an attribute to point to the compiled bytecode file? -Barry

5 6

Re: [Python-Dev] PEP 3147: PYC Repository Directories
by Barry Warsaw Feb. 7, 2010

Feb. 7, 2010

On Feb 01, 2010, at 02:04 PM, Paul Du Bois wrote: >It's an interesting challenge to write the file in such a way that >it's safe for a reader and writer to co-exist. Like Brett, I >considered an append-only scheme, but one needs to handle the case >where the bytecode for a particular magic number changes. At some >point you'd need to sweep garbage from the file. All solutions seem >unnecessarily complex, and unnecessary since in practice the case >should not come up. I don't think that part's difficult. The byte code's only going to change if the source file has changed, and in that case, /all/ the byte code in the "fat pyc" file will be invalidated, so the whole thing can be deleted by the first writer. I'd worked that out in the original fat pyc version of the PEP. -Barry

2 2

Re: [Python-Dev] PEP 3147: PYC Repository Directories
by Barry Warsaw Feb. 7, 2010

Feb. 7, 2010

On Feb 01, 2010, at 08:26 AM, Tim Delaney wrote: >The pyc/pyo files are just an optimisation detail, and are essentially >temporary. Given that, if they were to live in a single directory, to me it >seems obvious that the default location for that should be in the system >temporary directory. I an immediately think of the following advantages: > >1. No one really complains too much about putting things in /tmp unless it >starts taking up too much space. In which case they delete it and if it gets >reused, it gets recreated. IIUC the Filesystem Hierarchy Standard correctly, then these files really should go under /var/cache/python. (Don't ask me where that would be on non-FHS compliant systems <cough>Windows</cough>). I've explained in other followups why I don't particularly like separating the source from the cache files though, but if you wanted a sick approach: Take the full absolutely path to the .py file, plus the magic number, plus the time stamp and hash that. Cache the pyc file under /var/cache/python/<hash>. -Barry

1 0

Re: [Python-Dev] PEP 3147: PYC Repository Directories
by Barry Warsaw Feb. 7, 2010

Feb. 7, 2010

On Jan 31, 2010, at 08:10 PM, Silke von Bargen wrote: >Martin v. Löwis schrieb: >> There is also the issue of race conditions with multiple simultaneous >> accesses. The original format for the PEP had race conditions for >> multiple simultaneous writers; ZIP will also have race conditions for >> concurrent readers/writers (as any new writer will have to overwrite >> the central directory, making the zip file temporarily unavailable - >> unless they copy it, in which case we are back to writer/writer >> races). >> >> Regards, >> Martin >> >> >Good point. OTOH the probability for this to happen actually is very small. And yet, when it does happen, it's probably a monster to debug and defend against. Unless we have a convincing cross-platform story for preventing these race conditions, I think a single-file (e.g. zipfile) approach is infeasible. -Barry

1 0