about line numbers

Just curious: Is python with vs. without "-O" equivalent today regarding line numbers? Are SET_LINENO opcodes a plus in some situations or not? Next, I see quite often several SET_LINENO in a row in the beginning of code objects due to doc strings, etc. Since I don't think that folding them into one SET_LINENO would be an optimisation (it would rather be avoiding the redundancy), is it possible and/or reasonable to do something in this direction? A trivial example:
3 SET_LINENO 2 6 SET_LINENO 3 9 LOAD_CONST 1 (1) 12 STORE_FAST 0 (a) 15 LOAD_CONST 2 (None) 18 RETURN_VALUE
Can the above become something like this instead: 0 SET_LINENO 3 3 LOAD_CONST 1 (1) 6 STORE_FAST 0 (a) 9 LOAD_CONST 2 (None) 12 RETURN_VALUE -- Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252

[Vladimir Marangozov]
In theory it should make no difference, except that the trace mechanism makes a callback on each SET_LINENO, and that's how the debugger implements line-number breakpoints. Under -O, there are no SET_LINENOs, so debugger line-number breakpoints don't work under -O. I think there's also a sporadic buglet, which I've never bothered to track down: sometimes a line number reported in a traceback under -O (&, IIRC, it's always the topmost line number) comes out as a senseless negative value.
All opcodes consume time, although a wasted trip or two around the eval loop at the start of a function isn't worth much effort to avoid. Still, it's a legitimate opportunity for provable speedup, even if unmeasurable speedup <wink>. Would be more valuable to rethink the debugger's breakpoint approach so that SET_LINENO is never needed (line-triggered callbacks are expensive because called so frequently, turning each dynamic SET_LINENO into a full-blown Python call; if I used the debugger often enough to care <wink>, I'd think about munging in a new opcode to make breakpoint sites explicit). immutability-is-made-to-be-violated-ly y'rs - tim

Tim Peters wrote:
[Vladimir Marangozov, *almost* seems ready to give up on a counter- productive dict pessimization <wink>]
Of course I will! Now everything is perfectly clear. Thanks.
... So, that's why <wink>.
Now, *this* one explanation of yours should go into a HowTo/BecauseOf for developers. I timed your scripts and a couple of mine which attest (again) the validity of the current implementation. My patch is out of bounds. It even disturbs from time to time the existing harmony in the results ;-) because of early resizing. All in all, for performance reasons, dicts remain an exception to the rule of releasing memory ASAP. They have been designed to tolerate caching because of their dynamics, which is the main reason for the rare case addressed by my patch. -- Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252

[going back a week here, to dict resizing ...] [Vladimir Marangozov]
Yes, except I don't think there is such a rule! The actual rule is a balancing act between the cost of keeping memory around "just in case", and the expense of getting rid of it. Resizing a dict is extraordinarily expensive because the entire table needs to be rearranged, but lists make this tradeoff too (when you del a list element or list slice, it still goes thru NRESIZE, which still keeps space for as many as 100 "extra" elements around). The various internal caches for int and frame objects (etc) also play this sort of game; e.g., if I happen to have a million ints sitting around at some time, Python effectively assumes I'll never want to reuse that int storage for anything other than ints again. python-rarely-releases-memory-asap-ly y'rs - tim

Tim Peters wrote:
[going back a week here, to dict resizing ...]
Yes, and the subject line does not correspond to the contents because at the moment I've sent this message, I ran out of disk space and the mailer picked a random header after destroying half of the messages in this mailbox.
Good point.
Yes, and I'm somewhat sensible to this issue afer spending 6 years in a team which deals a lot with memory management (mainly DSM). In other words, you say that Python tolerates *virtual* memory fragmentation (a funny term :-). In the case of dicts and strings, we tolerate "internal fragmentation" (a contiguous chunk is allocated, then partially used). In the case of ints, floats or frames, we tolerate "external fragmentation". And as you said, Python tolerates this because of the speed/space tradeoff. Hopefully, all we deal with at this level is virtual memory, so even if you have zillions of ints, it's the OS VMM that will help you more with its long-term scheduling than Python's wild guesses about a hypothetical usage of zillions of ints later. I think that some OS concepts can give us hints on how to reduce our virtual fragmentation (which, as we all know, is a not a very good thing). A few kewords: compaction, segmentation, paging, sharing. We can't do much about our internal fragmentation, except changing the algorithms of dicts & strings (which is not appealing anyways). But it would be nice to think about the external fragmentation of Python's caches. Or even try to reduce the internal fragmentation in combination with the internal caches... BTW, this is the whole point of PyMalloc: in a virtual memory world, try to reduce the distance between the user view and the OS view on memory. PyMalloc addresses the fragmentation problem at a lower level of granularity than an OS (that is, *within* a page), because most Python's objects are very small. However, it can't handle efficiently large chunks like the int/float caches. Basically what it does is: segmentation of the virtual space and sharing of the cached free space. I think that Python could improve on sharing its internal caches, without significant slowdowns... The bottom line is that there's still plenty of room for exploring alternate mem mgt strategies that fit better Python's memory needs as a whole. -- Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252

Tim Peters wrote:
Could you elaborate a bit more on this? Do you mean setting breakpoints on a per opcode basis (for example by exchanging the original opcode with a new BREAKPOINT opcode in the code object) and use the lineno tab for breakpoints based on the source listing? -- Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252

[Vladimir Marangozov]
In theory it should make no difference, except that the trace mechanism makes a callback on each SET_LINENO, and that's how the debugger implements line-number breakpoints. Under -O, there are no SET_LINENOs, so debugger line-number breakpoints don't work under -O. I think there's also a sporadic buglet, which I've never bothered to track down: sometimes a line number reported in a traceback under -O (&, IIRC, it's always the topmost line number) comes out as a senseless negative value.
All opcodes consume time, although a wasted trip or two around the eval loop at the start of a function isn't worth much effort to avoid. Still, it's a legitimate opportunity for provable speedup, even if unmeasurable speedup <wink>. Would be more valuable to rethink the debugger's breakpoint approach so that SET_LINENO is never needed (line-triggered callbacks are expensive because called so frequently, turning each dynamic SET_LINENO into a full-blown Python call; if I used the debugger often enough to care <wink>, I'd think about munging in a new opcode to make breakpoint sites explicit). immutability-is-made-to-be-violated-ly y'rs - tim

Tim Peters wrote:
[Vladimir Marangozov, *almost* seems ready to give up on a counter- productive dict pessimization <wink>]
Of course I will! Now everything is perfectly clear. Thanks.
... So, that's why <wink>.
Now, *this* one explanation of yours should go into a HowTo/BecauseOf for developers. I timed your scripts and a couple of mine which attest (again) the validity of the current implementation. My patch is out of bounds. It even disturbs from time to time the existing harmony in the results ;-) because of early resizing. All in all, for performance reasons, dicts remain an exception to the rule of releasing memory ASAP. They have been designed to tolerate caching because of their dynamics, which is the main reason for the rare case addressed by my patch. -- Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252

[going back a week here, to dict resizing ...] [Vladimir Marangozov]
Yes, except I don't think there is such a rule! The actual rule is a balancing act between the cost of keeping memory around "just in case", and the expense of getting rid of it. Resizing a dict is extraordinarily expensive because the entire table needs to be rearranged, but lists make this tradeoff too (when you del a list element or list slice, it still goes thru NRESIZE, which still keeps space for as many as 100 "extra" elements around). The various internal caches for int and frame objects (etc) also play this sort of game; e.g., if I happen to have a million ints sitting around at some time, Python effectively assumes I'll never want to reuse that int storage for anything other than ints again. python-rarely-releases-memory-asap-ly y'rs - tim

Tim Peters wrote:
[going back a week here, to dict resizing ...]
Yes, and the subject line does not correspond to the contents because at the moment I've sent this message, I ran out of disk space and the mailer picked a random header after destroying half of the messages in this mailbox.
Good point.
Yes, and I'm somewhat sensible to this issue afer spending 6 years in a team which deals a lot with memory management (mainly DSM). In other words, you say that Python tolerates *virtual* memory fragmentation (a funny term :-). In the case of dicts and strings, we tolerate "internal fragmentation" (a contiguous chunk is allocated, then partially used). In the case of ints, floats or frames, we tolerate "external fragmentation". And as you said, Python tolerates this because of the speed/space tradeoff. Hopefully, all we deal with at this level is virtual memory, so even if you have zillions of ints, it's the OS VMM that will help you more with its long-term scheduling than Python's wild guesses about a hypothetical usage of zillions of ints later. I think that some OS concepts can give us hints on how to reduce our virtual fragmentation (which, as we all know, is a not a very good thing). A few kewords: compaction, segmentation, paging, sharing. We can't do much about our internal fragmentation, except changing the algorithms of dicts & strings (which is not appealing anyways). But it would be nice to think about the external fragmentation of Python's caches. Or even try to reduce the internal fragmentation in combination with the internal caches... BTW, this is the whole point of PyMalloc: in a virtual memory world, try to reduce the distance between the user view and the OS view on memory. PyMalloc addresses the fragmentation problem at a lower level of granularity than an OS (that is, *within* a page), because most Python's objects are very small. However, it can't handle efficiently large chunks like the int/float caches. Basically what it does is: segmentation of the virtual space and sharing of the cached free space. I think that Python could improve on sharing its internal caches, without significant slowdowns... The bottom line is that there's still plenty of room for exploring alternate mem mgt strategies that fit better Python's memory needs as a whole. -- Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252

Tim Peters wrote:
Could you elaborate a bit more on this? Do you mean setting breakpoints on a per opcode basis (for example by exchanging the original opcode with a new BREAKPOINT opcode in the code object) and use the lineno tab for breakpoints based on the source listing? -- Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252
participants (2)
-
Tim Peters
-
Vladimir Marangozov