[ python-Bugs-1750076 ] Python 2.5+ skips while statements in debuggers

[SourceForge.net]

Bugs item #1750076, was opened at 2007-07-09 09:24
Message generated for change (Comment added) made by ncoghlan
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1750076&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: None
Group: Python 2.5
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Chris Lasher (gotgenes)
>Assigned to: Neal Norwitz (nnorwitz)
Summary: Python 2.5+ skips while statements in debuggers

Initial Comment:
Starting with Python 2.5, debuggers pass through a "while" statement on its declaration, but never return to it during the remainder of the loop. They should be able to return to the "while" statement as it has an evaluative step in it. This has several important implications: one may not check the state of variables at prior to their evaluation in the "while" statement, and any breakpoints set on the "while" statement are ignored once within the while loop.

Python prior versions (2.4 and below) exhibit expected behavior in that the "while" statement is returned to after each successful iteration through the loop, and breakpoints on "while" statements are honored.

----------------------------------------------------------------------

>Comment By: Nick Coghlan (ncoghlan)
Date: 2007-08-01 00:48

Message:
Logged In: YES 
user_id=1038590
Originator: NO

It's a genuine bug in the AST compiler - there are some workarounds in
compile.c to make sure a for loop gets visited properly in the debugger,
but there is no equivalent for while loops at this stage.

I've had a quick look, but wasn't able to apply the same trick to while
loops - hopefully Neal will have some insight into the matter.

----------------------------------------------------------------------

Comment By: Nir (nir1408)
Date: 2007-07-24 22:01

Message:
Logged In: YES 
user_id=933518
Originator: NO

Hi Chris,

I have done some digging and it seems the problem is rooted in the way
co_lnotab is constructed for code objects
(http://docs.python.org/ref/types.html#l2h-139).

Consider for example the following code:

> line 0: def loop():
> line 1:    while True:
> line 2:        pass

Both Python 2.4 and Python 2.5 emit the same disassembly for that code
(using dis.dis()) :

>  1            0 SETUP_LOOP              12 (to 15)
>         >>    3 LOAD_GLOBAL              0 (True)
>               6 JUMP_IF_FALSE            4 (to 13)
>               9 POP_TOP
>
>  2           10 JUMP_ABSOLUTE            3
>         >>   13 POP_TOP
>              14 POP_BLOCK
>         >>   15 LOAD_CONST               0 (None)
>              18 RETURN_VALUE

However there is a difference with the co_lnotab structure between the
versions.
Python 2.4 emits the following offsets:

> byte code offsets:   [0, 3, 7]
> source code offsets: [1, 0, 1]

Python 2.5 emits the following offsets for the same code:

> byte code offsets:   [0, 10]
> source code offsets: [1, 1]

Note: The above output was generated with the following code:

> print 'byte code offsets:  ', [ord(x) for x in
> loop.func_code.co_lnotab[::2]]
> print 'source code offsets:', [ord(x) for x in
> loop.func_code.co_lnotab[1::2]]

So, what happens is that in offset 10, the while loop jumps back with
JUMP_ABSOLUTE 3, but in Python 2.5 that offset is missing from the byte
code offsets in co_lnotab. Python considers only offsets that appear in
that structure for the purpose of tracing, and therefore that line
(source line number 1) is skipped when jumped back to in Python 2.5

co_lnotab is constructed in compile.c but since that file was changed
considerably between Python 2.4 and Python 2.5 I did not try to pinpoint
the source of difference.

In addition I do not know the motivation of this difference. It could be
a mere bug, but it can also be by design, meaning that simply reverting
the co_lnotab construction behavior back might break Python 2.5
elsewhere.

Cheers,
Nir 

----------------------------------------------------------------------

Comment By: Chris Lasher (gotgenes)
Date: 2007-07-09 09:37

Message:
Logged In: YES 
user_id=1180453
Originator: YES

Notice how in Python 2.4, the breakpoint is honored through each iteration
through the while loop.

--

chris at feathers:~/development/playground$ python2.4 -m pdb simple.py 
> /home/chris/development/playground/simple.py(3)?()
-> a = 10
(Pdb) b 5
Breakpoint 1 at /home/chris/development/playground/simple.py:5
(Pdb) l
  1     #!/usr/bin/env python
  2  
  3  -> a = 10
  4  
  5 B   while a > 0:
  6         a -= 1      # how do I check the value of a at end of last
iter.?
  7  
  8     print "Fin!"
[EOF]
(Pdb) r
> /home/chris/development/playground/simple.py(5)?()
-> while a > 0:
(Pdb) p a
10
(Pdb) r
> /home/chris/development/playground/simple.py(5)?()
-> while a > 0:
(Pdb) p a
9
(Pdb) r
> /home/chris/development/playground/simple.py(5)?()
-> while a > 0:
(Pdb) p a
8
(Pdb) 


----------------------------------------------------------------------

Comment By: Chris Lasher (gotgenes)
Date: 2007-07-09 09:35

Message:
Logged In: YES 
user_id=1180453
Originator: YES

Notice how in Python 2.5, the breakpoint is met once then passed over,
leading straight to the final print statement

--

chris at feathers:~/development/playground$ python2.5 -m pdb simple.py 
> /home/chris/development/playground/simple.py(3)<module>()
-> a = 10
(Pdb) b 5
Breakpoint 1 at /home/chris/development/playground/simple.py:5
(Pdb) l
  1     #!/usr/bin/env python
  2  
  3  -> a = 10
  4  
  5 B   while a > 0:
  6         a -= 1      # how do I check the value of a at end of last
iter.?
  7  
  8     print "Fin!"
[EOF]
(Pdb) r
> /home/chris/development/playground/simple.py(5)<module>()
-> while a > 0:
(Pdb) r
Fin!
--Return--
> /home/chris/development/playground/simple.py(8)<module>()->None
-> print "Fin!"
(Pdb) 


----------------------------------------------------------------------

Comment By: Chris Lasher (gotgenes)
Date: 2007-07-09 09:32

Message:
Logged In: YES 
user_id=1180453
Originator: YES

Here's a file for demonstration purposes.
File Added: simple.py

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1750076&group_id=5470