[ python-Bugs-1628987 ] inspect trouble when source file changes

[SourceForge.net]

Bugs item #1628987, was opened at 2007-01-05 13:43
Message generated for change (Comment added) made by philipdumont
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1628987&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
Group: Python 2.4
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: phil (philipdumont)
Assigned to: Nobody/Anonymous (nobody)
Summary: inspect trouble when source file changes

Initial Comment:
backtrace (relevant outer frames only):

  File "/path/to/myfile", line 1198, in get_hook_name
    for frame_record in inspect.stack():
  File "/usr/mbench2.2/lib/python2.4/inspect.py", line 819, in stack
    return getouterframes(sys._getframe(1), context)
  File "/usr/mbench2.2/lib/python2.4/inspect.py", line 800, in getouterframes
    framelist.append((frame,) + getframeinfo(frame, context))
  File "/usr/mbench2.2/lib/python2.4/inspect.py", line 775, in getframeinfo
    lines, lnum = findsource(frame)
  File "/usr/mbench2.2/lib/python2.4/inspect.py", line 437, in findsource
    if pat.match(lines[lnum]): break
IndexError: list index out of range

Based on a quick look at the inspect code, I think
this happens when you:

  - Start python and load a module
  - While it's running, edit the source file for the
    module (before inspect tries to look into it).
  - Call a routine in the edited module that will
    lead to a call to inspect.stack().

During an inspect.stack() call, inspect will open
source files to get the source code for the
routines on the stack.  If the source file that is
opened doesn't match the byte compiled code that's
being run, there are problems.  Inspect caches the
files it reads (using the linecache module), so if
the file gets cached before it is edited, nothing
should go wrong.  But if the source file is edited
after the module is loaded and before inspect has
a chance to cache the source, you're out of luck.

Of course, this shouldn't be a problem in production
code, but it has bit us more than once in a
development environment.

Seems like it would be easy to avoid by just comparing
the timestamps on the source/object files.  If the
source file is newer, just behave the same as if it
wasn't there.

Attached is a stupid little python script that
reproduces the problem.



----------------------------------------------------------------------

>Comment By: phil (philipdumont)
Date: 2007-03-09 13:25

Message:
Logged In: YES 
user_id=1364034
Originator: YES

What I had in mind was something like changing:

    def getsourcefile(object):
        """Return the Python source file an object was defined in, if it
exists."""
        filename = getfile(object)
        if string.lower(filename[-4:]) in ['.pyc', '.pyo']:
            filename = filename[:-4] + '.py'
        # ...the rest of the function...

to

    def getsourcefile(object):
        """Return the Python source file an object was defined in, if it
exists."""
        filename = getfile(object)
        if string.lower(filename[-4:]) in ['.pyc', '.pyo']:
            src_filename = filename[:-4] + '.py'
            try:
                # if src file has been edited since we loaded the obj
file, ignore it
                if os.stat(filename).st_mtime >
os.stat(src_filename).st_mtime:
                    filename = src_filename
            except OSError:
                pass
        # ...the rest of the function...


But now that I've tried to implement it, and thought about it a bit more,
I see some
of the issues with what I thought would be a simple fix:

- I originally thought that, if a module is loaded from source (either
because the
objfile did not exist, or because objfile was older), and the compiled src
was successfully
written to a new objfile, that inspect.getfile() would return the path to
the objfile.
I see now that that is not the case; it returns the srcfile.  That makes
my fix
a bit more difficult -- now you have to find out if there is an obj file,
and if
so, what it's called.

And even if you had a handle on both the srcfile and the objfile:

- Even if the srcfile's timestamp is newer than the objfile's, it doesn't
necessarily
mean that the srcfile has been edited since it was loaded.  It could be
that the srcfile
was already newer than the objfile before the module was loaded, so the
module was
loaded from the srcfile, and the objfile was not updated because it was
not writable by
the user.

- Even if the srcfile's timestamp is older than the objfile's, it doesn't
necessarily
mean that the srcfile represents what you have loaded in memory.  It could
be that
after you loaded the module, your colleague down the hall edited the
source, loaded
the module, and in the process was successful in updating the objfile. 
(Not a likely
scenario, but...)

- If the module was loaded from source, there was no objfile, and an
objfile could not
be created because the directory was not writable, the fix doesn't help at
all.

To be able to fix this right, you'd have to know what the timestamp was on
whatever file
was loaded at the time that import loaded it.  I guess import itself would
have to stash
this away somewhere.


----------------------------------------------------------------------

Comment By: Collin Winter (collinwinter)
Date: 2007-03-08 16:19

Message:
Logged In: YES 
user_id=1344176
Originator: NO

Could you possibly work up a patch demonstrating exactly what you're
proposing with the "comparing the timestamps" solution? That seems like a
lot of complication for such a rare issue, but I'd be interested in seeing
a patch all the same.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1628987&group_id=5470