[Python-ideas] Re: Faster object representation for UIs

Hello, On Sat, 25 Jul 2020 16:34:16 +0300 Elizabeth Shashkova wrote:

Hi!

Thanks everyone for the interest and for the suggested options! I would like ro add my two coins and clarify some moments as the original requester of this feature.

1. We need this lazy `__repr__` calculation inside our debugger, where we work with different user's objects.

Did you consider that calling a __repr__ on an object may start formatting your (or user's) hard drive (arbitrary code is executed)? Alternatively, that may take a long time (maybe it mines bitcoin on each call), as you discovered. So, the behavior of calling __repr__ arbitrarily doesn't seem to be a good approach for a *generic* Python debugger. Then probably you don't write a generic Python debugger, but an adhoc, special-purpose one. And the problems you're facing is with specific object types, and it may be a good idea to contact projects which supply those types and tell them that you special-purpose debugger has problems with them. I still wonder what chance may be that the answer would be "don't call __repr__ then". Speaking of technical side, it's indeed sad at times that Python's __str__/__repr__ require to materialize entire representation in memory. More frugal approach would be: def __stream_repr__(self, stream): stream.write("") This can be a great economy of memory, for implementations which care about that (is that CPython?). That actually could provide means to address the posed problem, as "stream" can be a custom stream object, whose .write() method checks e.g. representation size or time budget, and if it's exceeded, throws an exception, which should be called at code which calls repr() in the first place. Other alternative could be turning __repr__ into generator (i.e., introducing __irepr__), but that's quite an expensive solution, given that generator instance need to be heap-allocated on each call, before it can be iterated over.

Usually it isn't some specific type, for which you know that it'll be big and its `__repr__` calculation will be slow (like, for example, pandas.DataFrame). Sometimes it can be just a composition of builtin types, like in the example below. On the top level it's just a `dict` with 10 elements and you don't expect its `repr()` to be slow, but it takes 13 secs on my machine to calculate it. ``` import time

def build_data_object(): data = dict() for i in range(10): temp_dict = dict() for j in range(10): temp_dict[str(j)] = "a" * 30000000 data[str(i)] = temp_dict return data

obj = build_data_object() start = time.time() repr(obj) finish = time.time() print("Time: %.2f" % (finish - start)) ```

2. I also agree it isn't the best idea to add additional parameters to `repr` or `str`. Just a function like `lazy_repr` implemented in stdlib will be already very useful.

3. But I also believe this issue can't be solved without changes in the language or stdlib, because you can't predict the length of `repr` for an object of unknown type without calculation of the whole string. But I hope it should be possible to check current buffer size during `__repr__` generation and interrupt it when it reaches the limit. (sorry, I'm not a CPython developer and I might be too naive here, so correct me if I'm wrong).

Elizaveta Shashkova.

сб, 25 июл. 2020 г. в 13:27, Serhiy Storchaka :

...

24.07.20 18:10, Gábor Bernát пише:

...

I would like to have a lazy repr evaluation for the objects! Sometimes users have many really large objects, and when debugger is trying to show them in Variables View (=show their string representation) it can takes a lot of time. We do some tricks, but they not always work. It would be really-really cool to have parameter in repr, which defines max number of symbols we want to evaluate during repr for this object. Maybe repr is not the best here, because that should be interpreter meaningful, but instead the __str__ method that's better for this. Maybe we could pass in an optional limit argument to these methods, so that the user can decide what to print depending on how many characters he has left?

Any takes, better ideas how we could help this problem?

We need a structural repr protocol, which would represent complex object as a structure containing items and attributes, so pprint() would know how to format a multiline text representation, and graphic tools could represent objects as a tree, with collapsed by default deep children and long sequences which can be expanded interactively. It was discussed in the past, but we still do not have good specification of such protocol. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/SDW6AH... Code of Conduct: http://python.org/psf/codeofconduct/

-- Best regards, Paul mailto:pmiscml@gmail.com