On Tue, Oct 2, 2018 at 8:44 PM, David Teresi <dkteresi@gmail.com> wrote:

print(f'{value!d}') is a lot of symbols and boilerplate to type out just for a debugging statement that will be deleted later. Especially now that breakpoint() exists, I can't really see myself using this.

I also don't see the use case of it being within an f-string, because I've never had to interpolate a debug string within some other string or format it in a fancy way. You said it yourself, taking advantage of other f-string features isn't very useful in this case.

If other people can find a use for it, I'd suggest making it ita own function -- debug(value) or something similar.

There was some discussion of this back in April: https://mail.python.org/pipermail/python-ideas/2018-April/050113.html I think the way I'd do it would be: Step 1: Take the current "lnotab" that lets us map bytecode offsets -> line numbers, and extend it with more detailed information, so that we can map e.g. a CALL operation to the exact start and end positions of that call expression in the source. This is free at runtime, and would allow more detailed tracebacks (see [1] for an example), and more detailed coverage information. It would definitely take some work to thread the necessary information through the compiler infrastructure, but I think this would be a worthwhile feature even without the debug() use case. Step 2: Add a 'debug' helper function that exploits the detailed information to reconstruct its call, by peeking into the calling frame and finding the source for the call. Of course this would be a strange and ugly thing to do for a regular function, but for a debugging helper it's reasonable. So e.g. if you had the code: total = debug(x) + debug(y / 10) The output might be: debug:myfile.py:10: 'x' is 3 debug:myfile.py:10: 'y / 10' is 7 Or if you have a clever UI, like in an IDE or ipython, maybe it overrides the debug() operator to print something like: total = debug(*x*) + debug(y / 10) ^ *3* total = debug(x) + debug(*y / 10*) ^^^^^^^ *7* (for anyone for whom the rendering is borked: on my screen the "x" on the first line and the "y / 10" on the second line are highlighted in a different font, and the carets draw an underline beneath them.) -n [1] https://mail.python.org/pipermail/python-ideas/2018-April/050137.html -- Nathaniel J. Smith -- https://vorpus.org

On Wed, Oct 3, 2018, 03:55 Eric V. Smith <eric@trueblade.com> wrote:

On 10/3/2018 1:40 AM, Nathaniel Smith wrote:

...

I think the way I'd do it would be:

Step 1: Take the current "lnotab" that lets us map bytecode offsets -> line numbers, and extend it with more detailed information, so that we can map e.g. a CALL operation to the exact start and end positions of that call expression in the source. This is free at runtime, and would allow more detailed tracebacks (see [1] for an example), and more detailed coverage information. It would definitely take some work to thread the necessary information through the compiler infrastructure, but I think this would be a worthwhile feature even without the debug() use case.

Step 2: Add a 'debug' helper function that exploits the detailed information to reconstruct its call, by peeking into the calling frame and finding the source for the call. Of course this would be a strange and ugly thing to do for a regular function, but for a debugging helper it's reasonable. So e.g. if you had the code:

total = debug(x) + debug(y / 10)

The output might be:

debug:myfile.py:10: 'x' is 3 debug:myfile.py:10: 'y / 10' is 7

I'm not positive, but isn't this what q does?

The difference is that without "step 1", there's no reliable way to figure out the value's source text. q does it by grabbing the source line and making some guesses based on heuristics, but e.g. in the example here it gets confused and prints: 0.0s <module>: x) + q(y / 10=3 0.0s <module>: x) + q(y / 10=7 So you can think of this idea as (1) make it possible to implement a reliable version of q, (2) add an built-in implementation. -n

On Wed, Oct 3, 2018 at 2:11 PM Anders Hovmöller <boxed@killingar.net> wrote:

...

This would be used in debugging print statements, that currently end up looking like:

print(f'value={value!r}')

and would now be:

print(f'{value!d}')

It seems to me that a short form for keyword arguments would improve this situation too. So instead of your suggestion one could do:

print(dict(=value))

And of course this feature wouldn’t be a minor feature on f-strings but a feature that is generally useful and composable so the above could be improved:

def debug(**kwargs): for k, v in kwargs.items(): print(f’{k}={v}’)

debug(=value, =another)

What if it's not a simple name, though? The OP gave this (somewhat simplistic, but indicative) example: print(f'next: {value+1!d}') AIUI, keyword arguments are all supposed to be legal names/atoms, so you aren't supposed to do something like this: debug(**{"value+1":value+1}) even though it does work in current versions of CPython. So even if your "=value" syntax did permit it, I wouldn't want to guarantee that in the language. (Side point: Please watch your mailer. The debug() function above has smart quotes in it, which means it can't be copied and pasted into the interpreter. Not a big problem with trivially-simple functions, but if it's something larger, it's annoying to have to track down "SyntaxError: invalid character in identifier" to figure out why it's not doing what you think it is.) ChrisA

On Wed, Oct 03, 2018 at 06:57:19AM +0200, Anders Hovmöller wrote:

debug(next=value+1)

Still shorter than the proposed syntax

Are we trying to emulate Perl now? *wink*

and much more readable.

So you say. To me that looks like a regular function call, which calls an ordinary function "debug" and takes a simple keyword argument next with value "value+1". Things which contain compiler magic should look special, not like ordinary function calls.

...

AIUI, keyword arguments are all supposed to be legal names/atoms, so you aren't supposed to do something like this:

debug(**{"value+1":value+1})

Really? That seems pretty weird to me. I’ve used that type of thing in production code from time to time.

The fact that this works is, I think, an accident of implementation: py> def spam(**kw): ... print(kw) ... py> spam(**{"value+1": 42}) {'value+1': 42} rather than a guaranteed language feature. I can't find any relevent documentation on it, but I'd be very wary about relying on it. (To be honest, I expected it to fail until I tried it.) You certainly can't do this: py> spam(value+1=42) File "<stdin>", line 1 SyntaxError: keyword can't be an expression -- Steve

...

and much more readable.

So you say.

To me that looks like a regular function call, which calls an ordinary function "debug" and takes a simple keyword argument next with value "value+1".

That was what it was. That was not intended to be about magic, that was the normal case where you didn't want the magic. I think you might be taking a habit of interpreting my mails in the least flattering way due to our previous disagreements. I hope we can put this behind us going forward.

Things which contain compiler magic should look special, not like ordinary function calls.

Good. I agree :P

...

...

AIUI, keyword arguments are all supposed to be legal names/atoms, so you aren't supposed to do something like this:

debug(**{"value+1":value+1})

Really? That seems pretty weird to me. I’ve used that type of thing in production code from time to time.

The fact that this works is, I think, an accident of implementation:

py> def spam(**kw): ... print(kw) ... py> spam(**{"value+1": 42}) {'value+1': 42}

rather than a guaranteed language feature. I can't find any relevent documentation on it, but I'd be very wary about relying on it.

(To be honest, I expected it to fail until I tried it.)

I don't really think accidents of implementation are different from hard requirements in Python, as it applies to alternative implementations. In practice if it deviates from CPython then it's broken. There is no language spec, there is only CPython. This has been the experience and approach of PyPy as far as I've understood it after having followed their blog over the years. / Anders

...

I don't really think accidents of implementation are different from hard requirements in Python, as it applies to alternative implementations. In practice if it deviates from CPython then it's broken. There is no language spec, there is only CPython. This has been the experience and approach of PyPy as far as I've understood it after having followed their blog over the years.

Definitely not true. There have been times when other implementors have come to python-dev and said, hey, is this part of the spec or is it an implementation detail? And the answer determines whether they care about that or not. For just a few examples:

1) Reference counting vs nondeterministic garbage collection 2) O(1) indexing/slicing of Unicode strings 3) "\N{...}" string escapes (okay, that's standardized, but optional) 4) Reuse of id() values 5) The "slots" mechanism for dunder method lookup

The language spec determines, in some cases, that a CPython implementation detail has been promoted to standard. More often, it determines that other Pythons are permitted to behave differently.

Sometimes they will come away from this list thinking they don't care but then their users will report bugs over and over again and they'll just have to do it anyway. You probably won't hear about most of those. Trees that fall in the forest when no one is there do in fact make a sound. / Anders

As a user I would be really pleased with the change proposed, as I use extensively use f-string in my logging (the fact that I have to evaluate the string whatever the logger level is is not a performance hit for my application), and usually the most readable logging format is something akin to f"interesting_variable_name={interesting_variable_name}, big_list[:10]={big_list[:10]}". 2018-10-03 11:17 GMT+02:00 Chris Angelico <rosuav@gmail.com>:

On Wed, Oct 3, 2018 at 7:09 PM Anders Hovmöller <boxed@killingar.net> wrote:

...

...

...

I don't really think accidents of implementation are different from

...

...

...

Definitely not true. There have been times when other implementors have come to python-dev and said, hey, is this part of the spec or is it an implementation detail? And the answer determines whether they care about that or not. For just a few examples:

1) Reference counting vs nondeterministic garbage collection 2) O(1) indexing/slicing of Unicode strings 3) "\N{...}" string escapes (okay, that's standardized, but optional) 4) Reuse of id() values 5) The "slots" mechanism for dunder method lookup

The language spec determines, in some cases, that a CPython implementation detail has been promoted to standard. More often, it determines that other Pythons are permitted to behave differently.

Sometimes they will come away from this list thinking they don't care but then their users will report bugs over and over again and they'll just have to do it anyway. You probably won't hear about most of those. Trees

hard requirements in Python, as it applies to alternative implementations. In practice if it deviates from CPython then it's broken. There is no language spec, there is only CPython. This has been the experience and approach of PyPy as far as I've understood it after having followed their blog over the years. that fall in the forest when no one is there do in fact make a sound.

...

And sometimes, people all over the world learn to write "with open(...) as f:" instead of just letting f expire. There IS a language spec, and pretending there isn't one doesn't change that.

ChrisA _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- -- *Nicolas Rolin* | Data Scientist + 33 631992617 - nicolas.rolin@tiime.fr <prenom.nom@tiime.fr> *15 rue Auber, **75009 Paris* *www.tiime.fr <http://www.tiime.fr>*

The present discussion thread is spreading into other areas, such as what constitutes the Python language specification.

I let myself get side tracked. This will be my last mail discussing that off topic thread. Sorry about that.

I'd like to contribute to a broader discussion, of how we can improve the debugging experience. But I feel uncomfortable doing so in a discussion thread that started with your admirably clear focus.

Please reconsider! We need more view points and good ideas. / Anders

On 10/3/2018 5:29 AM, Jonathan Fine wrote:

Hi Eric

Summary: This email is mainly about process. One discussion thread or several. I think the decision is yours.

You wrote suggesting an enhancement for debugging:

...

Here’s the idea: for f-strings, we add a !d conversion operator, which is superficially similar to !s, !r, and !a. The meaning of !d is: produce the text of the expression (not its value!), followed by an equal sign, followed by the repr of the value of the expression.

You gave as one of several examples that given value = 10 the expression f'{value!d}' evaluates to 'value=10' which can then be printed or logged.

I thank you and Larry Hastings for coming up with and exploring this idea.

Here's my personal opinion, which does not count for much. I like the output and ease of use. I have doubts about the implementation, and how it might work with other things. And also, I'd like the change to go through the PEP process.

I don't think a simple f-string feature needs a PEP. The implementation is pretty simple, since the f-string parser already has access to the string.

The present discussion thread is spreading into other areas, such as what constitutes the Python language specification. I'd like to contribute to a broader discussion, of how we can improve the debugging experience. But I feel uncomfortable doing so in a discussion thread that started with your admirably clear focus.

Briefly, Eric, would you like the general discussion of improving debugging to take place in this thread, or in another.

I think it would be more productive as a separate thread.

Finally, if I may I thank you and Larry for your work on this important topic, and for starting this useful discussion.

Thanks. It's what we do! Eric

On 10/3/2018 7:07 AM, Eric V. Smith wrote:

On 10/3/2018 12:23 AM, Tim Peters wrote:

...

[Eric V. Smith <eric@trueblade.com <mailto:eric@trueblade.com>>]

    Here’s the idea: for f-strings, we add a !d conversion operator, which     is superficially similar to !s, !r, and !a. The meaning of !d is:     produce the text of the expression (not its value!), followed by an     equal sign, followed by the repr of the value of the expression. ...

    The result is a string, so if you really wanted to, you could use a     string formatting spec. So:

    print(f'*{value!d:^20}*'

    would produce:

    *      value=10      *

    Although I don’t think that would be very useful in general.

Me neither ;-)  But what if

     {EXPR!d:FMT}

acted like the current

     EXPR={EXPR:FMT}

?  I'd find _that_ useful often.  For example, when displaying floats, where the repe is almost never what I want to see.

 >>> f"math.pi={math.pi:.2f}" 'math.pi=3.14'

I have plenty of code already embedding stuff of the form

     EXPR {EXPR:FMT}

and don't really care whether there's a space or an "=" between the chunks.  "!d" could act like a macro-expansion operator automating a mechanical transformation inside the f-string.  Then I could read "!d" as "duplicate" instead of as "debug" ;-)

That's a very interesting idea. It breaks the expectation (which might be mine alone!) that the format spec is used on the result of the conversion specifier (!s, !r). But maybe that's okay. It does seem generally more useful than the format spec being used on the repr of the evaluated expression.

After giving this some more thought, the problem with this approach is that there's no way to get the repr of the object, which for debugging can be pretty useful (think strings). Remember, by default object.__format__ calls object.__str__. I guess we could get around this by combining !d and !r and assigning some meaning to that, which I'd rather not do. Or, this occurred to me right before hitting "send": if there's no format spec, use repr(expr). If there is one (even if it's zero-length), use format(expr, spec). I'll have to think on that for a while. Maybe there's too much voodoo going on there. Eric

Gah! You are overthinking it. This idea is only worth it if it's dead simple, and the version that Eric posted to start this thread, where !d uses the repr() of the expression, is the only one simple enough to bother implementing. -- --Guido van Rossum (python.org/~guido)

On 10/3/2018 8:47 PM, Tim Peters wrote:

Note that transforming

   {EXPR!d:FMT}

into

   EXPR={repr(EXPR):FMT}

is actually slightly more involved than transforming it into

   EXPR={EXPR:FMT}

so I don't buy the argument that the original idea is simpler.  More magical and less useful, yes ;-)

Actually, my proposal is to apply FMT to the entire result of EXPR={repr(EXPR)}, not just the repr(EXPR) part. I'm not sure either is particularly useful. Eric

On Oct 4, 2018, at 6:24 PM, Tim Peters <tim.peters@gmail.com> wrote:

[Tim]

...

...

Note that transforming

{EXPR!d:FMT}

into

EXPR={repr(EXPR):FMT}

is actually slightly more involved than transforming it into

EXPR={EXPR:FMT}

so I don't buy the argument that the original idea is simpler. More magical and less useful, yes ;-)

[Eric V. Smith]

...

Actually, my proposal is to apply FMT to the entire result of EXPR={repr(EXPR)}, not just the repr(EXPR) part. I'm not sure either is particularly useful.

I think I’ve come around to Tim’s way of thinking on this. If the format spec is basically useless (which I think it would be if it applied to either the whole string or to the repr of the value), then what’s the point of allowing it? I also want the 90% case to be addressed by just plain repr. I did consider (and implement) a version that prohibited format specs, but it seemed wrong to remove a feature that might have some clear use. So, my compromise is: - f"{expr!d}" expands to f"expr={repr(expr)}", but - f"{expr!d:spec} expands to f"expr={format(expr, spec)}" There’s some more discussion here: https://bugs.python.org/issue36774#msg341306 As Paul Moore says on that issue, this is explained as: use repr, unless you specify a format spec, then use format.

BTW, I checked, and I've never used !r, !s, or !a. So the idea that the format could apply to a string - when EXPR itself doesn't evaluate to a string - is simply foreign to me. I suppose it's natural to people who do use ![rsa] all the time - if such people exist ;-)

Especially when debugging and logging, I use !r all the time. Eric

...

Here’s the idea: for f-strings, we add a !d conversion operator, which is superficially similar to !s, !r, and !a. The meaning of !d is: produce the text of the expression (not its value!),

I SO WANT THIS AS A GENERAL FEATURE, not just for f-strings, it hurts.

Actually what I want is an executable object (a function?) which has the AST and text of the expression attached. If putting this into f-strings is a first step towards getting this thunk-like thing, then I don't need to read any further, I'm +10000 :-)

Well... this is a trivial expansion of the keyword argument short form proposal you were very strongly opposed ^_- So, I suggested `foo(=a)` as short form for `foo(a=a)`, but if we interpret it instead as `foo(=a)` -> `foo(**{'a': a}`, which is really the same thing for most purposes, then we could trivially get `foo(=1+3*bar)` -> 'foo(**{'1+3*bar': 1+3*bar})' I'm fine with this proposal. And thanks Chris for the idea! / Anders

On 10/3/2018 3:54 AM, Steven D'Aprano wrote:

On Tue, Oct 02, 2018 at 08:27:03PM -0400, Eric V. Smith wrote:

...

Here’s the idea: for f-strings, we add a !d conversion operator, which is superficially similar to !s, !r, and !a. The meaning of !d is: produce the text of the expression (not its value!),

I SO WANT THIS AS A GENERAL FEATURE, not just for f-strings, it hurts.

Actually what I want is an executable object (a function?) which has the AST and text of the expression attached. If putting this into f-strings is a first step towards getting this thunk-like thing, then I don't need to read any further, I'm +10000 :-)

I feel your pain, but we're a long way from that.

...

produces:

value=10 next: value+1=11 s='a string!'

I can see lots of arguments about whether the equals sign should have spaces around it.

Maybe !d for no spaces and !D for spaces?

print(f'next: {value+1!d}') print(f'next: {value+1!D}')

would print

next: value+1=11 next: value+1 = 11

I'm not sure it's worth the hassle. I considered a whole format spec language for this, but it sort of got out of hand.

...

I’m not proposing this for str.format(). It would only really make sense for named arguments, and I don’t think print('{value!d}'.format(value=value) is much of a win.

I'm not so sure that it only makes sense for named arguments. I think it works for arbitrary expressions too:

f'{len("NOBODY expects the Spanish Inquisition!")!d}'

ought to return

'len("NOBODY expects the Spanish Inquisition!")=39'

which I can see being very useful in debugging expressions.

Yes, that's part of the proposal, but I wasn't very clear. That's what the "value+1" example was supposed to convey. My comment about named parameters applied only to str.format(), and why it won't be supported there. Eric