On Mon, Feb 8, 2016 at 6:36 PM Mahmoud Hashemi wrote:

Without something like this, there's no way to ask an object if it's repr is of manageable length, or how its repr could be meaningfully shortened. Usually I just chop out the middle and add an ellipsis, but as for the time spent generating that middle, I'll never get those cycles back.

With the addition of ``__short_repr__`` would there be a change to the default Python REPL? If not, how would you use the new magic method? If so, then why not just change __repr__ for all the built-ins to truncate automatically? As far as I know, reprs are not part of the backwards compatibility guarantee. (Note, I'm not advocating for changing reprs.) If the usage would be to write a function that checks for the existence of __short_repr__, then why not simply move the implementation of ``if len(s) < maxlen`` etc. to that function?

Mahmoud Hashemi wrote:

I was curious if what kind of interest there would be in adding an overridable method to types to allow getting back a repr, but a predictably short one.

If you just want to know the type and identity of an object, you can use object.__repr__(x). -- Greg

On 9 Feb 2016, at 02:49, Mahmoud Hashemi wrote:

Roundtrippable reprs are certainly part of Python canon, whether or not they are universally used (Chris), or guaranteed (Mike).

They can be part of the canon all they want, but if they’re not universally guaranteed then I don’t know that this is a real problem. It means that the world of Python objects divides into two kinds: first, those with __repr__ return values that can be round tripped and those with __repr__ return values that cannot be round tripped. Given that objects of the second type already exist (I know this for a fact because I have written some quite recently!), it would be an error to assume that the identity 'eval(repr(x)) == x’ holds for arbitrary types. In fact, not only does it not hold for all third-party types, it doesn’t even hold for all built-in types:

...

...

x = float(‘nan’) repr(x) ‘nan’ eval(repr(x)) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<string>", line 1, in <module> NameError: name 'nan' is not defined

I think the reality is that there is no constraint on the representation of arbitrary types to be round-trippable in any way. Again, all custom types have non-round-trippable representations by default, many more eclectic built-in types have non-round-tripppable representations (in addition to NaN, the memoryview object leaps to mind). I can also note the Python documentation on repr:

For many types, this function makes an attempt to return a string that would yield an object with the same value when passed to eval(), otherwise the representation is a string enclosed in angle brackets that contains the name of the type of the object together with additional information often including the name and address of the object.

If the language doesn’t even try to enforce the idea that representations will be round-trippable, I think there’s just no problem here. Cory

On Feb 9, 2016, at 00:56, Cory Benfield wrote:

...

On 9 Feb 2016, at 02:49, Mahmoud Hashemi wrote:

Roundtrippable reprs are certainly part of Python canon, whether or not they are universally used (Chris), or guaranteed (Mike).

They can be part of the canon all they want, but if they’re not universally guaranteed then I don’t know that this is a real problem. It means that the world of Python objects divides into two kinds: first, those with __repr__ return values that can be round tripped and those with __repr__ return values that cannot be round tripped.

So? One use of round-trippable reprs is to copy them from output and paste them into source code or an interactive session. When doing so, you almost always know that you're dealing with a builtin or third-party type that's round-trippable--and, when you're surprised, it's almost always obvious, because you get something that looks nothing at all like the source equivalent, and that raises a SyntaxError if you try to evaluate it anyway. And even those "almost"s aren't a problem in practice. Sure, a list that recursively contains itself looks misleadingly round-trippable, and will evaluate successfully into the wrong thing. But this rarely comes up in practice--and, if it does, because there's a human being inspecting, debugging, or playing with things, rather than a program, it's easy to deal with. This isn't theoretical--I do this all the time when debugging code, I write my own types to make them easier to debug this way, and it saves me time and hassle. In fact, it's one of the benefits of using Python over some of the other languages I use, where repr or toDebugString or whatever is never useful, instead of being usually useful especially with common types. The only thing the non-universality of round-tripping means is that you can't use repr with eval as a persistence format. Which is a good thing--you *shouldn't* use it as a persistence format, and that would be true even if it did work. But using it as an inspecting/debugging format is not a problem, and breaking that would be a bad idea. In fact, breaking it would make repr nearly pointless. Except for a types that define __repr__ but block __str__ with a TypeError (which is rare, and it's debatable whether those types are even valid), when would you ever use repr otherwise?

On 09.02.2016 12:13, Cory Benfield wrote:

...

On 9 Feb 2016, at 10:13, Andrew Barnert wrote:

So?

One use of round-trippable reprs is to copy them from output and paste them into source code or an interactive session. When doing so, you almost always know that you're dealing with a builtin or third-party type that's round-trippable--and, when you're surprised, it's almost always obvious, because you get something that looks nothing at all like the source equivalent, and that raises a SyntaxError if you try to evaluate it anyway.

And even those "almost"s aren't a problem in practice. Sure, a list that recursively contains itself looks misleadingly round-trippable, and will evaluate successfully into the wrong thing. But this rarely comes up in practice--and, if it does, because there's a human being inspecting, debugging, or playing with things, rather than a program, it's easy to deal with.

This isn't theoretical--I do this all the time when debugging code, I write my own types to make them easier to debug this way, and it saves me time and hassle. In fact, it's one of the benefits of using Python over some of the other languages I use, where repr or toDebugString or whatever is never useful, instead of being usually useful especially with common types.

The only thing the non-universality of round-tripping means is that you can't use repr with eval as a persistence format. Which is a good thing--you *shouldn't* use it as a persistence format, and that would be true even if it did work. But using it as an inspecting/debugging format is not a problem, and breaking that would be a bad idea.

In fact, breaking it would make repr nearly pointless. Except for a types that define __repr__ but block __str__ with a TypeError (which is rare, and it's debatable whether those types are even valid), when would you ever use repr otherwise? I don’t think we’re arguing the same point. I’m not saying that __repr__ shouldn’t be round-trippable: if that makes sense for your type then totally fine, go for it. However, I am saying that I don’t see the advantage in having *both* a round-trippable and non-round-trippable repr for the same type. If you’re copying and pasting then the length of the round-trippable representation is a non-issue, and you actively want it to appear in your debug output rather than a shorter version that elides information required to round-trip the data. So in this case, what purpose does the shorter version serve?

Same question here. I don't see much use for __short_repr__ being worth the hassle of maintaining possibly two implementations of almost the same thing. All in all, I don't think __repr__ needs optimization in the first place. Best, Sven

For those examples, the pprint/reprlib behavior probably wouldn't be much different than dict/list of tuples. I can put my vote in, but I prefer to defer design specifics to the creators/maintainers of those types. That's my point. Right now I feel like repr() is a powerful, but underleveraged function in the builtin ecosystem. Instead of pprint.pformat(), why not add a keyword argument, and have repr(obj, pretty=True), and have that call through with arguments to a more full featured __repr__? Same could go for repr(obj, length=150, depth=2). These could have defaults set for REPL sessions. Everyone is doing hacky, incomplete heuristic introspection, from IPython to the standard library. Instead, let's embrace the complexities of viewing application state, and allow for formalization of community efforts. Mahmoud On Wed, Feb 10, 2016 at 11:18 AM, Michael Selik wrote:

The appropriate truncation for any repr that is too long would depend on the particulars of both the object and the purpose of the output. If reprlib and pprint were improved for say, defaultdict and OrderedDict, or more generic things, what would its features be?

I'm guessing you wouldn't just duplicate the functionality of ``pprint(dict(x))``.

On Tue, Feb 9, 2016, 4:09 AM Mahmoud Hashemi wrote:

...

If you want to make the case that default reprs should be non-roundtrippable in the case that they're too long, that's a fine and separate discussion. Though I have argued in the past that float('nan') would be a less surprising/more useful repr.

And that's what it's about, usefulness. It's useful to be able to copy in and out of the REPL, even for very large reprs. I've done it, my coworkers and students do it, and you've probably done it as well.

But there are other times (often outside the REPL), where that is not the case, and being able to address them explicitly, in the vein of reprlib and pprint -- but better -- would be *useful*. Who hasn't wished that the built-in defaultdict and OrderedDict were as pprintable or reprlib.repr-able as dict.

There's plenty of room to improve.

On Tue, Feb 9, 2016 at 12:56 AM, Cory Benfield wrote:

...

...

On 9 Feb 2016, at 02:49, Mahmoud Hashemi wrote:

Roundtrippable reprs are certainly part of Python canon, whether or not they are universally used (Chris), or guaranteed (Mike).

They can be part of the canon all they want, but if they’re not universally guaranteed then I don’t know that this is a real problem. It means that the world of Python objects divides into two kinds: first, those with __repr__ return values that can be round tripped and those with __repr__ return values that cannot be round tripped.

Given that objects of the second type already exist (I know this for a fact because I have written some quite recently!), it would be an error to assume that the identity 'eval(repr(x)) == x’ holds for arbitrary types. In fact, not only does it not hold for all third-party types, it doesn’t even hold for all built-in types:

...

...

...

x = float(‘nan’) repr(x) ‘nan’ eval(repr(x)) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<string>", line 1, in <module> NameError: name 'nan' is not defined

I think the reality is that there is no constraint on the representation of arbitrary types to be round-trippable in any way. Again, all custom types have non-round-trippable representations by default, many more eclectic built-in types have non-round-tripppable representations (in addition to NaN, the memoryview object leaps to mind).

I can also note the Python documentation on repr:

...

For many types, this function makes an attempt to return a string that would yield an object with the same value when passed to eval(), otherwise the representation is a string enclosed in angle brackets that contains the name of the type of the object together with additional information often including the name and address of the object.

If the language doesn’t even try to enforce the idea that representations will be round-trippable, I think there’s just no problem here.

Cory

On Wednesday, February 10, 2016 12:29 PM, Random832 wrote:

On Tue, Feb 9, 2016, at 03:56, Cory Benfield wrote:

...

I think the reality is that there is no constraint on the representation of arbitrary types to be round-trippable in any way. Again, all custom types have non-round-trippable representations by default, many more eclectic built-in types have non-round-tripppable representations (in addition to NaN, the memoryview object leaps to mind).

One other example is classes and functions though in many cases it's not clear why this should be the case. Have the default for top-level functions and classes check whether it's reachable through [module].[name] and if so return that. The default for methods could use [class].[name], or [obj].[name] for bound methods.

Instead of the current representation, you could have the default repr on objects use pickle.

That seems like a bad idea. First, as a human programmer, the repr "[1, 2, (3, 'spam')]" means something to me--and the same thing to me as to Python; the repr "unrepr('silly.C object at 0x106ec2b38', b'\x80\x03csilly\nC\nq\x00)\x81q\x01}q\x02X\x01\x00\x00\x00xq\x03K\x02sb.')" means less to me than the current "", while taking up a lot more space. Meanwhile, if I've changed the silly module since printing out the repr, that pickle will fail--or, worse, and more likely, succeed but give me the wrong value. And, since eval'ing reprs is generally something you do when experimenting or when debugging code that you're actively working on, this will be very common. Meanwhile, eval("[1, 2, (3, 'spam')]") == [1, 2, (3, 'spam')], and that's true for most containers and "value-type" objects, which tend to be the kinds of things that have round-trippable reprs today. That probably won't be true for instances of arbitrary types where the creator didn't think of a repr. Finally, when I paste a repr into the REPL or into my source and it needs further qualification, as with datetime or Decimal, I can almost always figure this out pretty easily. And I would much rather have to figure it out and then paste "decimal.Decimal('123.456')" into my source than have the computer figure it out and paste "__import__('decimal').Decimal('123.456')" or something with a pickle in the middle of it into my source. It's true that pasting reprs is just something that tends to often work when you'd expect it to today, not something guaranteed--but since repr is used for exploration and debugging by humans, not for automated processing by scripts, "tends to often work" tends to often be very good. So there really isn't a problem here. And your suggestion doesn't really help that use anyway; the only use it helps is using repr as an automated serialization format, which we explicitly _don't_ want to help. If you want automated serialization, just use pickle. There's no reason to twist repr and eval to support that use case almost but not quite as well (and even less securely and efficiently). If you want something more readable, use something like YAML with a custom type library, or jsonpickle, etc. Or, if you think we need a more human-readable pickle format, that might be an interesting idea, but there's no reason to twist repr into it, or to force it to be readable by eval instead of loads.

Michael: Yes, I have several projects on PyPI, some of which are related and somewhat popular, but I hardly think that a PyPI package can encompass the API changes I'm describing. We already have several built-in modules (reprlib, pprint) and 3rd party modules (IPython Notebook, Django 500 pages, the Werkzeug debugger) that see a great deal of usage, but only support a few builtinn types and have to hack heuristics for the rest. Cory: Introspecting application state happens at many levels, one step at a time. Not only is perfect the enemy of good here, but I really don't think an omni-solution like the one you're describing is a realistic, Pythonic goal. Besides, graphs are powerful, but don't go underestimating linear text. An enhanced repr allows developer-users to communicate the context of their usage so that the library authors can programmatically provide and prioritize information in their types. I've seen (and written) plenty of reprs that encode useful information in my types. To your example, I might have: . I still think you're selling repr short. It can be much more. One day we might have:

...

...

repr(obj, pretty=True, depth=2, width=120)

Though, as I said before, my primary use case isn't the REPL, and those arguments should be part of the REPL config (pprint defaults to width=80 for consoles). The point is that repr has room to evolve. Mahmoud On Thu, Feb 11, 2016 at 1:00 AM, Cory Benfield wrote:

...

On 10 Feb 2016, at 19:31, Mahmoud Hashemi wrote:

For those examples, the pprint/reprlib behavior probably wouldn't be much different than dict/list of tuples. I can put my vote in, but I prefer to defer design specifics to the creators/maintainers of those types. That's my point.

Right now I feel like repr() is a powerful, but underleveraged function in the builtin ecosystem. Instead of pprint.pformat(), why not add a keyword argument, and have repr(obj, pretty=True), and have that call through with arguments to a more full featured __repr__? Same could go for repr(obj, length=150, depth=2). These could have defaults set for REPL sessions.

Everyone is doing hacky, incomplete heuristic introspection, from IPython to the standard library. Instead, let's embrace the complexities of viewing application state, and allow for formalization of community efforts.

If we want to embrace the complexities of viewing application state, perhaps we should divorce this conversation from the repr altogether.

There’s plenty of applications where “viewing application state” will require more than the repr generally provides: you’ll want insight into private member variables, into the state of composed objects, and potentially into the state of some of those private member variables (e.g. is this lock held at this moment?). This is outside the generally accepted scope of the repr as I understand it: most people don’t provide the repr in this form.

If you want the complexity of application state then you probably want some way to dump the graph of objects starting with a specific one, along with their state. That’s totally beyond the scope of the repr, and is closer to what pickle would do for you (though as Andrew points out, pickle is not the right tool for this job either).

What I don’t understand is: if you are in the REPL already, what does any of this buy you? The REPL has plenty of powerful tools for introspecting Python objects: specifically, it has the Python programming language! If you want to introspect application state from the REPL, then just grab the object you want and start interrogating its state. dir() works, printing fields works, calling methods works, and all of this is far more powerful than anything you can get from the repr.

Cory

On Thu, Feb 11, 2016 at 01:24:42AM -0800, Mahmoud Hashemi wrote:

I still think you're selling repr short. It can be much more. One day we might have:

...

...

...

repr(obj, pretty=True, depth=2, width=120)

I'm sympathetic, but I'm not sure that repr() is the right place to be focusing. repr() is too closely tied to the REPL and the existing string formatting codes to start making radical changes. But we do have the pprint library, which (in my opinion) is underpowered and underused. I know that some people think that Python already has too many string conversion functions ( str()/__str__ and repr()/__repr__ ) but for my own use I've occasionally found that it has too few. Some time ago, I started working on a library for continued fractions. I count at least five useful/standard representations for a continued fraction. All of these would be equivalent: ContinuedFraction(1, 2, 3, 4, 5) [1; 2, 3, 4, 5] 1 + 1/(2 + 1/(3 + 1/(4 + 1/5))) 1 1 1 1 1 + ----- ----- ----- ----- 2 + 3 + 4 + 5 1 1 + ----------------- 1 2 + ------------- 1 3 + --------- 1 4 + ----- 5 (Apart from the first, the remaining four are standard notations used in mathematics.) Now obviously I can just add my own string conversion methods to the class, but it would be nice to be able to integrate with the pprint (and maybe reprlib) libraries in some way rather than an ad hoc set of arbitrarily named methods? I must admit that I'm just thinking aloud here, I don't actually have a concrete idea. But maybe my use-case will spark an idea in someone else. -- Steve

On Thu, Feb 11, 2016, at 05:59, Steven D'Aprano wrote:

Some time ago, I started working on a library for continued fractions. I count at least five useful/standard representations for a continued fraction. All of these would be equivalent:

(Apart from the first, the remaining four are standard notations used in mathematics.)

How about 1+\dfrac 1{2+\dfrac 1{3+\dfrac 1{4+\dfrac 1 5}}}? A way to get a "mathematical abstract syntax tree" that can be operated on and transformed into that or any of your notations [1+1/(2+1/(3+1/(4+1/5))) might be the default] for numeric objects (and other things such as matrices and vectors) might be nice.