Mailman 3 August 2020 - Python-ideas

Fwd: Re: Experimenting with dict performance, and an immutable dict
by Marco Sulla Oct. 15, 2020

Oct. 15, 2020

On Wed, 22 Jul 2020 at 15:35, Antoine Pitrou <solipsis(a)pitrou.net> wrote: > The deltas also look small for a micro-benchmark. I certainly don't > think this is a sufficient reason to add a new datatype to Python. > I think some of the optimizations can be experimented with dict itself. Deltas seem to be small, because I divide them by the number of timeit runs. I think that a 30-45% gain in constructor speed is not something small. The benchmarks in question have the Name that starts with "constructor". PS: sorry for double mail, I sent this message to only you by mistake. On Wed, 22 Jul 2020 at 18:26, Guido van Rossum <guido(a)python.org> wrote: > Did you study PEP 416 (frozendict) and PEP 603 (frozenmap)? > Yes. About frozenmap, at the very start I added to the benchmarks also immutables.Map, but I removed it when I realized that it was slower than frozendict in every bench. Maybe this is because immutables.Map is a C extension and not a builtin type. About PEP 416, yes, I tried to follow it. Indeed hash is calculated using the strategy described in the PEP. I also take a look to frozenset and tuple code. Frankly I must admit that the rejection of PEP 416 was a good idea. Indeed I started to implement an immutable dict only for a symmetry reason, and because I am somewhat fascinated by functional programming and immutables, without a rational reason. I suppose I would have given up and admitted that a frozendict is quite useless in Python, if I did not see that the constructor speed can be faster. I'm not sure 100% of the result. This is why I'm suggesting to try to implement the speed optimizations to the constructor of dict first: 1. the optimizations could be not safe. Indeed recently I got a segfault that I have to investigate 2. maybe dict can be really optimized further. After that, the difference between dict and frozendict performance could be minimal That said, maybe there are four good use cases for a frozendict: 1. they could be used for __annotations__ of function objects, and similar cases 2. they could be used to implement "immutable" modules or classes 3. frozendict can be easily cached, like tuples. 4. as I said, as an alternative to MappingProxyType, since it's very slow and it's used also in the C code of CPython and in the Python stdlib. This maybe is not useful because: a. the speed of MappingProxyType can be improved b. MappingProxyType can't be replaced because you *really* want and need a proxy

6 20

Decorators for class non function properties
by redradist＠gmail.com Sept. 14, 2020

Sept. 14, 2020

Hi all, Seems like this topic was previously raised, but what if we add possibility to decorate non function properties in class: ```python class Neuron: @linear_activation activation ```

17 71

Access (ordered) dict by index; insert slice
by Hans Ginzel Sept. 14, 2020

Sept. 14, 2020

Date: Fri, 26 Jun 2020 18:47:44 +0200 From: Hans Ginzel <hans(a)matfyz.cz> To: Hans Ginzel <hans(a)artax.karlin.mff.cuni.cz> Subject: Access (ordered) dict by index; insert slice Hello, thank you for making dict ordered. Is it planned to access key,value pair(s) by index? See https://stackoverflow.com/a/44687752/2556118 for example. Both for reading and (re)writing? Is it planned to insert pair(s) on exact index? Or generally to slice? See splice() in Perl, https://perldoc.perl.org/functions/splice.html. Use case: Represent database table metadata (columns). It is useful as to access columns both by name and by index as to insert column on specific position, https://dev.mysql.com/doc/refman/8.0/en/alter-table.html, “ALTER TABLE ADD COLUMN [FIRST |AFTER col]” (consider default order or table storage size optimisation by aligning). Thank you in advance, Hans PS1: Named tuples cannot be used, are immutable. PS2: See https://metacpan.org/pod/perlref#Pseudo-hashes:-Using-an-array-as-a-hash

32 182

Compound with .. else statement
by Jimmy Thrasibule Sept. 9, 2020

Sept. 9, 2020

Hi, In Python, there are multiple [compound statements](https://docs.python.org/3/reference/compound_stmts.html) with the `else` keyword. For example: ``` for x in iterable: if x == sentinel: break else: print("Sentinel not found.") ``` or: ``` try: do_something_sensitive() except MyError: print("Oops!") else: print("We're all safe.") ``` In my situation, I would like to mix the `with` statement with `else`. In this case I would like that if no exception is raised within the `with` to run the `else` part. For example: ``` with my_context(): do_something_sensitive() else: print("We're all safe.") ``` Now imagine that in my `try .. except` block I have some heavy setup to do before `do_something_sensitive()` and some heavy cleanup when the exception occurs. I'd like my context manager to do the preparation work, execute the body, and cleanup. Or execute my else block only if there is no exception. Is there already a way to accomplish this in Python or can this be a nice to have? Regards, Jimmy

6 10

Changing item dunder method signatures to utilize positional arguments (open thread)
by Ricky Teachey Sept. 3, 2020

Sept. 3, 2020

The following comment is from the thread about adding kwd arg support to the square bracket operator (eg, `Vec = Dict[i=float, j=float]`). On Tue, Aug 4, 2020, 2:57 AM Greg Ewing <greg.ewing(a)canterbury.ac.nz> wrote: On 4/08/20 1:16 pm, Steven D'Aprano wrote: > Why would we want to even consider a new approach to handling keyword > arguments which applies only to three dunder methods, `__getitem__`, > `__setitem__` and `__delitem__`, instead of handling keyword arguments > in the same way that every other method handles them? These methods are already kind of screwy in that they don't handle *positional* arguments in the usual way -- packing them into a tuple instead of passing them as individual arguments. I think this is messing up everyone's intuition on how indexing should be extended to incorporate keyword args, or even whether this should be done at all. -- Greg So here is the main question of this thread: Is there really not a backwards compatible, creative way a transition to positional args from a tuple in the item dunders could not be accomplished? It's not my intention to champion any specific idea here, just creating a specific space for ideas to be suggested and mulled over. There could be several reasons for changing the item dunder signatures. The immediate reason is making the intuition around how to add kwd arg support to square brackets more obvious and sane. A second reason is it might be more intuitive for users who have to learn and remember that multiple arguments to [ ] get packed into a tuple, but this doesn't happen anywhere else. Another reason: it could make writing code for specialized libraries that tend to abuse (for the good of us all!) item dunders, like pandas, much easier. Right now such libraries have to rely on their own efforts to break up a key: def __getitem__(self, key): try: k1, k2 = key except TypeError: raise TypeError("two tuple key required") But for regular function calls (as opposed to item getting) we get to write our signature however we want and rely on the language to handle all of this for us: def f(k1, k2): # no worries about parsing out the arguments ----------- One idea: change the "real" names of the dunders. Give `type` default versions of the new dunders that direct the call to the old dunder names. The new get and del dunders would have behavior and signatures like (I am including **__kwargs since that could be an option in the future) : def __getx__(self, /, *__key, **__kwargs): return self.__getitem__(__key, **__kwargs) def __delx__(self, /,, *__key, **__kwargs): del self.__delitem__(__key, **__kwargs) However the set dunder signature would be a problem, because to mirror the current behavior we end up writing what is now a syntax error: def __setx__(self, /, *__key, __value, **__kwargs): self.__setitem__(__key, __value, **__kwargs) The intended meaning above would be that the last positional argument gets assigned to __value. Maybe someone could suggest a way to fix this. Item getting, setting, deleting would call these new dunders instead of the old ones. I haven't thought through how to handle inheritance-- I'm sort of hoping someone smarter than me could come up with a way to solve that...: class My: def __getx__(self, my_arg, *args, my_kwarg, **kwargs): # the way I have written things this super call will cause a recursion error v = super().__getitem__(*args, **kwargs) return combine(my_arg, my_kwarg, v) By the way I'm not bike shedding on what these new dunder names should be, though we could probably do worse then getx, setx, and delx (nice and short!). ----------- Ok now this is the part where I get to wait for everyone smarter than me to show the errors of my ways/how naive I am. :)

16 71

PEP 472 - new dunder attribute, to influence item access
by Jonathan Fine Aug. 31, 2020

Aug. 31, 2020

SUMMARY Sequences and mappings are both similar and different. Let's introduce and use a new dunder attribute, to give item access appropriate behaviour. This proposal is based on an earlier thread - see Acknowledgments for URL. INTRODUCTION In Python, there are two sorts of builtin objects that have item access. They are mappings and sequences. They have different abstract base classes. Mappings and sequences have fundamental similarities, and also fundamental differences. To see an example of this, let's consider >>> x = dict() # Mapping >>> y = [None] # Sequence Consider now >>> x[0] = 'hi' >>> x[0] == 'hi' True >>> 0 in x, 'hi' in x (True, False) Compare it with >>> y[0] = 'hi' # Similar >>> y[0] == 'hi' True >>> 0 in y, 'hi' in y # Different (False, True) THE PROBLEM Not taking into account the fundamental differences between mappings and sequences can, I think, cause of difficulty when considering the semantics of >>> z[1, 2, a=3, b=4] If z is a sequence (or multi-dimensional array) then many of us would like to think of item access as a function call. In other words, ideally, >>> z[1, 2, a=3, b=4] >>> z.__getitem__(1, 2, a=3, b=4) are to be equivalent. But if z is a mapping, then perhaps ideally we'd like an object >>> key = K(1, 2, a=3, b=4) such that >>> z[1, 2, a=3, b=4] >>> z.__getitem__(key) are equivalent. PRESENT BEHAVIOUR At present >>> z[1, 2, a=3, b=4] roughly speaking calls >>> internal_get_function(z, 1, 2, a=3, b=4) where we have something like def internal_get_function(self, *argv, **kwargs): if kwargs: raise SyntaxError if len(argv) == 1: key = argv[0] else: key = argv type(self).__getitem__(self, key) PROPOSAL I think it will help solve our problem, to give Z = type(z) a new dunder attribute that either is used as the internal_get_function, or is used inside a revised system-wide internal_get_function. That way, depending on the new dunder attribute on Z = type(z), sometimes >>> z[1, 2, a=3, b=4] >>> z.__getitem__(1, 2, a=3, b=4) are equivalent. And sometimes >>> z[1, 2, a=3, b=4] is equivalent to >>> key = K(1, 2, a=3, b=4) >>> z.__getitem__(key) all depending on the new dunder attribute on Z = type(z). I hope this contribution helps. ACKNOWLEDGEMENTS I've had much valuable help from Ricky Teachey in preparing this message. I've also been influenced by his and others contributions to an earlier thread, which he started 3 weeks ago. https://mail.python.org/archives/list/python-ideas@python.org/thread/FFXXO5… -- Jonathan

17 109

argmax and argmin to python list
by Filipp Bakanov Aug. 31, 2020

Aug. 31, 2020

I'd like to propose adding argmax and argmin functions to the python list. These functions return the index of a maximum / minimum element of the list. Eg: a = [1, 4, 2, 3] print(a.argmax()) # 1 print(a.argmin()) # 0 It's a very popular request (based on stackoverflow https://stackoverflow.com/questions/16945518/finding-the-index-of-the-value… ), and currently there is no elegant way to find it. What do you think?

7 10

Option Read/Write Files without discarding comments
by Steve Barnes Aug. 29, 2020

Aug. 29, 2020

It occurs to me that there are many situations where files are human authored and can include comments but by default when python reads/modifies/writes those files by default the comments are lost. Some examples include configfile, json and even python itself with the special case that docstrings are read without being discarded. If we wish to generate a commented file, (such as an initial version of a config file), then file type specific code is needed but in general if we read in a file the comments are discarded – usually of course this is what we need but not if we are going to be modifying values are re-writing. In the case of files that are read and parsed into python objects we could quite easily have an option to make comments a docstring on the resulting objects and a write option to output the docstrings (marked as comments). It would be necessary to have a mechanism for preserving file level comments that are interleaved with values/objects and after the last value/object. Of course, in many cases, it would be nice to have find commented object and uncomment object, possibly comment out object methods but I still believe that even without such methods could be useful to be able to round trip files that include comments. Writing this has also sparked a thought of wouldn’t it be nice to have a format for python line comments where the object so commented gets a docstring from the line comment, e.g. class StockItem: “”” Represents an item available in store “”” lineno: str #! Catalogue Item Number in the format AA-nnnn-AAA : The humble docstring can be so useful and as far as I know any python object can carry one but currently base classes such as int, etc. Have read only __doc__ members. Of course I am probably going to get told off for 2 related but divergent elements in one thread! Steve Barnes Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10

4 3

Re: What about having a .get(index, default) method for arrays like we have for dicts?
by Christopher Barker Aug. 28, 2020

Aug. 28, 2020

Oops, look like I took this off-list by accident -- bringing it back on. On Fri, Aug 28, 2020 at 12:50 PM Guido van Rossum <guido(a)python.org> wrote: > IMO the bar for attempting to solve this is pretty high, since I can think > of no way to solve it without breaking old code that doesn't involve > introducing new names. > Exactly -- so we're kind of stuck with the existing ABCs for a good long while. Maybe that's OK, but it does mean we are choosing to freeze at least this part of Python. But I agree with you that it's problematic to add features to the builtins as well (like .get() to list and friends that aren't in the ABCs, so we're really freezing a lot of the language. Which may be a good thing, but on this list at least, I expect people wont like that :-) FWIW I don't think there's an exception for dunders -- if we were to add > Sequence.__new_dunder__ we'd still break code due to register(). > hmm -- I never actually used register(), and had always assumed that it did some checking -- but apparently not, so yes, anything registered with an ABC that something was added to could now be broken thanks for the tip. Though that's probably less breakage, but breakage all the same. -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

1 0

What about having a .get(index, default) method for arrays like we have for dicts?
by Daniel. Aug. 28, 2020

Aug. 28, 2020

When I need to traverse nested dicts, is a common pattern to do somedict.get('foo', {}).get('bar', {}) But there is no such equivalent for arrays, wouldn't be nice if we can follow somedict.get('foo', {}).get('bar', []).get(10) ... ? What I do in this case is surround with try/except IndexError and set some variable to None, or wrap the try/except block in a function. But this is noise in my opinion, I just want to follow the same reasoning that I have with dicts: I'm searching for some path deep nested, and want a default if not found. What do you think? Regards -- “If you're going to try, go all the way. Otherwise, don't even start. ..." Charles Bukowski

19 88