[ python-Bugs-1180193 ] broken pyc files

[SourceForge.net]

Bugs item #1180193, was opened at 2005-04-10 13:10
Message generated for change (Comment added) made by arigo
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1180193&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Interpreter Core
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Armin Rigo (arigo)
Assigned to: Nobody/Anonymous (nobody)
Summary: broken pyc files

Initial Comment:
In a number of situations, the .pyc files can become "corrupted" in a subtle way: the co_filename attribute of the code objects it contains become wrong.  This can occur if we move or rename directories, or if we access the same set of files from two different locations (e.g. over NFS).

This corruption doesn't prevent the .pyc files from working, but the interpreter looses the reference to the source file.  It causes trouble in tracebacks, in the inspect module, etc.

A simple fix would be to use the following logic when importing a .py file: if there is a corresponding .pyc file, in addition to checking the timestamp, check the co_filename attribute of the loaded object.  If it doesn't point to the original .py file, discard the code object and ignore the .pyc file.

Alternatively, we could force all co_filenames to point to the .py file when loading the .pyc file.

I'll write a patch for whichever alternative seems better.

----------------------------------------------------------------------

>Comment By: Armin Rigo (arigo)
Date: 2007-04-03 11:31

Message:
Logged In: YES 
user_id=4771
Originator: YES

If you ask me, I think that when the importing
system finds both a .py and a .pyc for a module,
then it should ignore all co_filename and replace
them with the real path of the .py file.  I can't
see any point of not doing so.

There are many other quirks caused by .pyc files
accidentally remaining around, but we cannot fix them
all as long as the .pyc files are at the same time
a cache for performance reason and a redistributable
program format (e.g. if "rm x.py" or "svn up" deletes
a .py file, then the module is still importable via
the .pyc left behind, a great way to oversee the fact
that imports elsewhere in the project need to be
updated).

----------------------------------------------------------------------

Comment By: Ziga Seilnacht (zseil)
Date: 2007-04-03 07:16

Message:
Logged In: YES 
user_id=1326842
Originator: NO

This problem is reported quite often in the tracker,
although it shows up in different places:

http://www.python.org/sf/1666807
http://www.python.org/sf/1051638

I closed those bugs as duplicates of this one.

The logging package is also affected:

http://www.python.org/sf/1669498
http://www.python.org/sf/1633605
http://www.python.org/sf/1616422


----------------------------------------------------------------------

Comment By: Armin Rigo (arigo)
Date: 2007-03-28 13:40

Message:
Logged In: YES 
user_id=4771
Originator: YES

What I called "corruption" is the situation
where both the .py and the .pyc files are
present, but the filename stored in the .pyc
co_filenames is no longer the valid absolute
path of the corresponding .py file, for any
reason (renaming, NFS views, etc.).

This situation causes the tracebacks and the
inspect module to fail to locate the .py file,
which I consider a bug.

----------------------------------------------------------------------

Comment By: Martin v. Löwis (loewis)
Date: 2007-03-28 12:01

Message:
Logged In: YES 
user_id=21627
Originator: NO

I fail to see the corruption. It is quite desirable and normal to only
ship pyc files - that the file name they refer to is actually present is
not a requirement at all.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1180193&group_id=5470