TITLE: Saving and sharing debug code PROBLEM: Sometimes debug-only code should be saved and shared. For example, one person's code written to solve a bug might be needed a second time, by someone else. Or perhaps the same person, again. This is particularly likely when the bug is difficult or stubborn. POSSIBLE SOLUTION At present, __debug__ is a keyword identifier, which can take only the values True and False. It gets its value on Python startup, and then its value can't be changed. The possible solution is to allow Python startup to give __debug__ an additional value, namely 2. Once this is done, code blocks such as if __debug__ >= 2: # my debug-only code goes here will effectively be ignored, except when requested. Python already does this for blocks such as if __debug__: # ignored if optimised code is being generated See also: https://docs.python.org/3/library/constants.html#__debug__ __debug__ This constant is true if Python was not started with an -O option. See also the assert statement.

Chris Angelico wrote:

Why not just discuss each proposal on its own merits, independently of any other proposals? Do they interact with each other?

Good question. I think they will interact. Not in terms of implementation, but in terms of benefit. To quote the Zen of Python:

There should be one-- and preferably only one --obvious way to do it. Although that way may not be obvious at first unless you're Dutch.

However, we won't know for sure until the proposals are in. -- Jonathan

TITLE: output debug information easily and quickly POSSIBLE SOLUTION Short form of keyword arguments where foo(=a, =1+bar) Is expanded at compile time to foo(**{'a': a, '1+bar': 1+bar}) Then we can just create a normal debug function: def debug_print(**vars): for k, v in vars.items(): print(f'{k}={v}') this is of course useful for many other things as Steven has pointed out. / Anders

...

foo(=a, =1+bar)

Unfortunately, that won't help with Jonathan's inital example expression "big_array[5:20]" as it's not a valid keyword.

I didn't understand that. The example you are referring to is print('big_array[5:20] =', big_array[5:20]) Right? Nothing is a keyword in that example or in my example. My suggestion is that we could do: my_func(=big_array[5:20]) And it would be compile time transformed into my_func(**{'big_array[5:20]': big_array[5:20]}) and then my_func is just a normal function: def my_func(**kwargs): Whatever It's a very simple textual transformation. / Anders

...

...

...

foo(=a, =1+bar)

...

Unfortunately, that won't help with Jonathan's inital example expression "big_array[5:20]" as it's not a valid keyword.

I didn't understand that. The example you are referring to is print('big_array[5:20] =', big_array[5:20])

Nothing is a keyword in that example or in my example. My suggestion is that we could do: my_func(=big_array[5:20])

And it would be compile time transformed into my_func(**{'big_array[5:20]': big_array[5:20]})

and then my_func is just a normal function:

def my_func(**kwargs): Whatever

You're right, that case will work. I was thinking of

In [1]: foo(a+b=1) File "<ipython-input-1-61f24dcb4c20>", line 1 foo(a+b=1) ^ SyntaxError: keyword can't be an expression

Sure. But no one suggested it either so I don't see how it's relevant. / Anders

Anders Hovmöller suggested

Short form of keyword arguments where foo(=a, =1+bar) Is expanded at compile time to foo(**{'a': a, '1+bar': 1+bar})

Chris Angelico wrote:

That is not guaranteed to work. In another thread it was pointed out that this is merely a CPython implementation detail, NOT a language feature.

Here's a variant of Anders' suggestion. First, here's a dict literal {'a':1, 'b': 2, 'c':3} and here's another way to write an equivalent dict dict(a=1, b=2, c=3) So how about extending Python so that, for example, {=(1 + bar), } is equivalent to {'1 + bar': 1 + bar, } The basic idea is Anders's, recast to avoid Chris's problem. Anders: Are you willing to accept this change, if need be? Chris: Please speak up, if you think this may depend on CPython. Off topic: https://data.grammarbook.com/blog/apostrophes/apostrophes-with-names-ending-... To show singular possession of a name ending in s or z, some writers add just an apostrophe. Others also add another s. -- Jonathan

Spun off as a new thread because this isn't really specific to debugging.

Thank you for starting a new thread, to discuss this idea. -- Jonathan

On Thu, Oct 4, 2018 at 4:36 AM Anders Hovmöller <boxed@killingar.net> wrote:

...

...

It's a very simple textual transformation.

That is not guaranteed to work. In another thread it was pointed out that this is merely a CPython implementation detail, NOT a language feature.

Pedantry. Ok. Let's be pedantic: it's a simple textual transformation AND a promotion of an implementation detail to a spec requirement. Happy?

Then I'm strongly -1 on it. Happy? :) ChrisA

So how about extending Python so that, for example, {=(1 + bar), } is equivalent to {'1 + bar': 1 + bar, }

The basic idea is Anders's, recast to avoid Chris's problem.

Chris' problem isn't an actual problem though. Its just a few sentences in a PEP. It might be a problem for other python implementations but I'm gonna put say 100 dollars on that it's not actually so. Pypy, jython, ironpython, who else? Without looking I'm betting they have the same implementation detail. Any takers on such a bet? One is not allowed to look before taking the bet but I can't check that so scout honor applies.

Anders: Are you willing to accept this change, if need be?

I would but I think it's worse in every possible way :P It looks like a set, not a dict for example. / Anders

On Thu, Oct 4, 2018 at 4:54 AM Anders Hovmöller <boxed@killingar.net> wrote:

...

...

Then I'm strongly -1 on it. Happy? :)

And In case it's not clear why I said that, btw: It's not mere pedantry.

Good to see you understood yourself why that mail wasn't so good.

...

By restating your proposal in those terms, you make it far broader than a simple textual transformation. Imagine if you'd said it like this:

"Okay, let's be pedantic. As well as my other proposals, I'm also requiring that you be able to use 'a+b' as a variable name."

That is most definitely not a simple proposal. And that means it should be discussed as a much much broader change: disconnecting keyword arguments from variable names. That should NOT just slide through as part of a separate change.

Imagine if I said something other totally irrelevant and that is bigger change indeed. But I didn't. I suggested not a change of CPython or PyPy or IronPython but a few sentences in a PEP. I also didn't suggest that it be snuck into the same PEP as my proposed syntax changes. I agree that would be bad. It should obviously be a separate PEP.

I'm not sure what you're calling irrelevant here. But sure. If you want to propose that, propose it. Start a new thread in which you propose that, as a language feature, kwargs are allowed to be invalid variable names.

I noticed you wouldn't take the bet too.

No, because the bet isn't the point. You're asking if any existing Python implementation assumes that kwargs are valid identifiers. I could easily create one and win the bet, and that still wouldn't be the point. You're proposing a change to the language specification, and that's not something to just gloss over. When PEP 572 started proposing changes to other semantics than just assignment expressions, there was a lot of pushback because that was seen as an independent proposal (even though it was fairly tightly bound to the assignment expressions themselves, in that it'd be extremely hard to observe the difference else). What you're proposing here is, similarly, a completely independent proposal, and not all that tightly bound. ChrisA

Hi Wolfram You tried

def f(a): .. print(a) f(**{"a":2}) 2 f(**{"a+1":2}) Traceback (most recent call last): File "python", line 1, in <module> TypeError: f() got an unexpected keyword argument 'a+1'

This is exactly what I would have expected. Please consider the following:

...

...

def f(a): pass f(**dict(b=1)) TypeError: f() got an unexpected keyword argument 'b'

Does CPython count as "other python implementation"?

Yes and No. Both are half correct. CPython is the reference implementation. Please also consider

...

...

def f(a, **kwargs): pass f(a=1, **{'a+1': 2})

...

...

f(a=1, **{(0, 1): 2}) TypeError: f() keywords must be strings

So far as I know, that a keyword be a string is the only constraint, at least in CPython. For example

...

...

def f(a, **kwargs): pass f(a=1, **{'': 2}) f(a=1, **{'def': 2})

So I think Anders proposal works in CPython. I think you forgot the **kwargs in the parameters to f. best regards Jonathan

...

Imagine if I said something other totally irrelevant and that is bigger change indeed. But I didn't. I suggested not a change of CPython or PyPy or IronPython but a few sentences in a PEP. I also didn't suggest that it be snuck into the same PEP as my proposed syntax changes. I agree that would be bad. It should obviously be a separate PEP.

I'm not sure what you're calling irrelevant here. But sure. If you want to propose that, propose it. Start a new thread in which you propose that, as a language feature, kwargs are allowed to be invalid variable names.

But wouldn't it make sense to have a motivating example? Like the one we're discussing? Not just suggest it out of the blue?

You're proposing a change to the language specification, and that's not something to just gloss over.

Many people are suggesting language spec changes in this thread and quite a few others. This is the forum for it.

When PEP 572 started proposing changes to other semantics than just assignment expressions, there was a lot of pushback because that was seen as an independent proposal (even though it was fairly tightly bound to the assignment expressions themselves, in that it'd be extremely hard to observe the difference else). What you're proposing here is, similarly, a completely independent proposal, and not all that tightly bound.

Sure. But I'm only proposing it in the "what if?" way. It's a discussion to see what other solutions exists for the problem that the thread started discussing. A You and me keep derailing. It's quite depressing. I don't want to be in a constant shouting game with you. I want to discuss ideas. / Anders

On Thu, Oct 04, 2018 at 04:17:41AM +1000, Chris Angelico wrote:

...

{=(1 + bar), } is equivalent to {'1 + bar': 1 + bar, }

What is so special about dicts that this only works there? If we're going to have syntax to capture the source (and AST) of an expression, it ought to work everywhere. And it ought to be callable, without having to call eval() on the source, which is slow. -- Steve

On Wed, Oct 3, 2018 at 10:48 AM, Chris Angelico <rosuav@gmail.com> wrote:

On Thu, Oct 4, 2018 at 2:30 AM Anders Hovmöller <boxed@killingar.net> wrote:

...

Nothing is a keyword in that example or in my example. My suggestion is that we could do:

my_func(=big_array[5:20])

And it would be compile time transformed into

my_func(**{'big_array[5:20]': big_array[5:20]})

and then my_func is just a normal function:

def my_func(**kwargs): Whatever

It's a very simple textual transformation.

That is not guaranteed to work. In another thread it was pointed out that this is merely a CPython implementation detail, NOT a language feature.

I'm curious where this is written down. Can you point to the relevant part of the language spec or pronouncement or whatever it was? -n -- Nathaniel J. Smith -- https://vorpus.org

Good morning, I read about a "time machine" debugger a long time ago. The debugger would collect all the information of all the calls and the programmer can just execute the code without breakpoints. Later, the programmer can follow the evolution of a variable until it reaches an erroneous value. I've never worked with that and it sounds really memory intensive, but the examples were quite interesting. When you are in a loop, you want to break at a certain iteration. When you are in a recursive function, you want to stop at the right point. etc. Cheers, Hans On 03/10/18 16:51, Jonathan Fine wrote:

This thread is about debugging. I suggest we start by collecting problems and possible solutions. And that after about a week of that, we start discussing what we've gathered.

We already have a problem and possible solution, provided by Eric Smith and Larry Hastings.

<begin> TITLE: f-string "debug" conversion URL: https://mail.python.org/pipermail/python-ideas/2018-October/053956.html PROBLEM Writing print('value = ', value) is tedious, particularly for more complicated expressions, such as print('big_array[5:20] =', big_array[5:20])

POSSIBLE SOLUTION

For f-strings, we add a !d conversion operator, which produces the text of an expression followed by its value. Thus, the two previous examples can be written more concisely as print(f'{value !d}') print(f'{big_array[5:20] !d}') </end>

I suggest for the moment that we just gather problem-solution pairs, much as above. I think they'll be between 5 and 15 such pairs. I'll post one to the current discussion thread in an hour or so.

And that after about a week, we move to discussing what we have. Finally, many thanks to Eric and Larry for their useful contribution to the important problem of debugging.

On Thu, Oct 4, 2018 at 6:33 PM Anders Hovmöller <boxed@killingar.net> wrote:

Ah, yes. Thank you. So it works in CPython 2.7. But I'm curious, does it work in very old versions? I'm not saying that this is important, because language changes always are for new versions. However, Anders' claim that this not a language change seemed too broad to me. It may be that this change has very little cost, but it should not be dismissed.

It works in:

Python 1 Python 2 Python 3 PyPy 6 IronPython Jython micropython

Are there more I should try?

I've no idea what you actually tried or what actually worked, since you haven't shown your code. However, it doesn't matter. This IS a language change, and it makes no difference how many implementations are lax enough to permit it currently. Citation: https://mail.python.org/pipermail/python-dev/2018-October/155437.html It's just like a program assuming that there will always be a user with UID 0, or assuming that every human being has a name that consists of a given name and a family name, or assuming that data sent across the internet will arrive unchanged. You can show millions, billions, trillions of examples that support your assumption, but that doesn't make any difference - the assumption is false as soon as there is a single counter-example, or as soon as the specification is shown to *permit* a counter-example. ChrisA

On 04/10/18 13:27, Anders Hovmöller wrote: [I think >> = ChrisA]

...

However, it doesn't matter. Of course it matters. It's the difference between changing the spec and changing the spec AND some implementation. There is a difference between those two things. You might not care but that's another topic.

In terms of defining specs, not there isn't a difference between those things. Changing the language spec is not a thing to do lightly regardless of how many implementations happen to do what you already want. While gather evidence that the chaos your change will cause is minimal is a good thing, it doesn't actually lower the bar for getting a language spec change through. Think of it this way: you are proposing to break a promise. Even if it's only a little thing and you're fairly sure everyone will forgive you, that's still not something you should ever want to get comfortable doing. -- Rhodri James *-* Kynesim Ltd

Hello all I think we've established enough basic facts for now, about using a non-identifier string as a keyword in a kwargs dictionary. I'd be most grateful if discussion of this particular topic could be suspended for now, to be resumed when it again becomes relevant. If you are keen to discuss this particular topic right now, please start a new discussion thread for that purpose. Later today, I'll be posting another problem and proposed solution to this thread, and would like your help on that. Many thanks Jonathan

On 04/10/18 18:41, Jonathan Fine wrote:

TITLE: PROBLEM: Debug print() statements cause doctests to fail Adding debug print(...) statements to code can cause doctests to fail. This is because both use sys.stdout as the output stream.

POSSIBLE SOLUTION: Provide and use a special stream for debug output. In other words, something like

...

...

...

import sys sys.stddebug = sys.stderr debug = lambda *argv, **kwargs: print(*argv, file=sys.stddebug, flush=True, **kwargs)

Note: Need to use current value of sys.stddebug, so can't use functools.partial. _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Or write your debug output to stderr? -- Rhodri James *-* Kynesim Ltd

I wrote:

Perhaps I've been too concise. If so, I apologise. My proposal is that the system be set up so that debug(a, b, c) sends output to the correct stream, whatever it should be.

See also: https://docs.python.org/3/whatsnew/3.7.html#pep-553-built-in-breakpoint

Python 3.7 includes the new built-in breakpoint() function as an easy and consistent way to enter the Python debugger.

So I'm suggesting a new built-in debug() function as an easy and consistent way to write out debugging information. -- Jonathan

Samuel Colvin wrote:

Python definitely needs a dedicated debug print command.

I've built python devtools with has such a command: https://github.com/samuelcolvin/python-devtools

Is this the kind of thing you were thinking about?

Thank you for this comment, Samuel. And also very much for your work on devtools. To answer your question: Yes, and No. I'm thinking of providing a builtin debug() command that 'does the right thing' according to the context. And the context would include the user's preferences. Which might be to use the print command in your devtools package. But the user might be in an IDE, which provides a separate window for the debug output. I suggest the builtin debug() would determine the API, and provide a reference implementation. And that many users (perhaps through their IDE) would choose a different implementation. -- Jonathan

Samuel Colvin writes:

Python definitely needs a dedicated debug print command.

For *commands* (ie, a REPL), you have Pdb already. It's unlikely a *statement* would be accepted, given that print was demoted from a statement to a function. And it's not obvious why a "debug.print()" function can't just as well be imported from a 3rd party library you get from PyPI. (There's already pdb.Pdb.prettyprint in the stdlib, though it may not be terribly easy to use outside of the Pdb REPL.)

(I'm not trying to promote devtools, just demonstrate what is possible.)

It seems to me that promoting devtools (unless you know a solution that's better for some reason) is exactly what you should do. Not that you should pretend it's already optimal. But it would be a concrete proposal that would serve to focus suggestions, and evaluate whether there is a single module that is sufficiently useful to enough people to deserve inclusion in the stdlib, or whether this is yet another case where a number of modules with varied feature sets available from PyPI is a better solution.

It would be difficult to fully integrate a package like this into the standard lib since it relies on pygments for colouring output, but not impossible.

I don't see why that's a problem, as long as the coloring feature is optional: have_color = None try: import pygments have_color = 'pygments' except ImportError: pass # or try other color libraries here and you don't even try to colorize if have_color is None. See the thread on xz and other optional external libraries that don't exist on some platforms and don't give nice errors when the import fails for the wrapper Python library. A bigger problem is if devtools does stuff on which there is no consensus that it's wanted in the stdlib. That "stuff" would need to be split out, which would possibly be a pain for you and any other existing users.