Python-ideas

python-ideas@python.org

79 participants
4841 discussions

PEP 9999 (provisional): support for indexing with keyword arguments

by Stefano Borini

PEP for support for indexing with keyword arguments is now submitted as PR. https://github.com/python/peps/pull/1612 Thanks to everybody involved in the development of the PEP and the interesting discussion. All your contributions have been received and often added to the document. If the PEP is approved, I would like to attempt an implementation, but I am not particularly skilled in the python internals. If a core developer is willing to teach me the ropes (especially in the new parser, I barely understood the syntax of the old one, but have no idea about the new one) and review my code I can give it a try. I would not mind refreshing my C a bit. -- Kind regards, Stefano Borini

10 minutes

Improved Error Message for "Unhashable Type"

by Samuel Freilich

The error message for using a mutable sequence as a set item or map key seems to frequently confuse Python beginners. The current wording is: >>> {[1, 2, 3]: [4, 5, 6]} TypeError: unhashable type: 'list' The first thing a Google search finds for "unhashable type" is ~4k Stack Overflow results like: https://stackoverflow.com/questions/13264511/typeerror-unhashable-type-dict The message does not include: * The type on which the operation is performed * Which input to the operation is the problem * The word "hashable" verbatim, which appears in the glossary * A link to https://docs.python.org/3/glossary.html#term-hashable * A suggestion of what to do instead For example: TypeError: dict keys must be hashable ( https://docs.python.org/glossary.html#term-hashable), 'list' is not. Consider using a primitive type (e.g. int, str) or immutable sequence (e.g. tuple, frozenset). (That could be too much stuff, my point is I think there's room for improvement over "unhashable type: 'list'".) I filed a bug about this (https://bugs.python.org/issue41114), but it was closed by a core contributor, who suggested discussing that on this list instead. While it's true that you can Google the error message and find the relevant information, I think the error message text would be worth improving. Peace, -Sam

2 hours, 26 minutes

A shortcut to load a JSON file into a dict : json.loadf

by The Nomadic Coder

Hi All, This is the first time I'm posting to this mailing group, so forgive me if I'm making any mistakes. So one of the most common ways to load json, is via a file. This is used extensively in data science and the lines. We often write something like :- with open(filename.json, "r") as f: my_dict = json.load(f) or my_dict = json.load(open("filename.json", "r")) Since this is sooooo common, why doesn't python have something like :- json.loadf("filename.json") Is there an obvious issue by defining this in the cpython? I don't whipping up a PR if it gains traction.

2 hours, 49 minutes

Adding mixed positional and keyword to PEP 637

by David Mertz

None of the examples posted in use cases show combinations of position and keyword indexing. This feels like an important example, and moreover without it, the discussion of what sentinel to use is less motivated. In discussions, I have mentioned two examples, e.g. distances[4:7, 100:120, 30:40, unit="metres"] Which could make comparisons of arrays with different internal units easier. disributed_array[4:7, 100:120, 30:40, source="worker1"] Which works be a natural way to address data that is *eventually consistent" in a cluster. But other examples are equally good. I think a soft change in an existing example makes it a better API: molecular_energy[4:7, 100:120, 30:40, BasisSet=Z3] These all amount to basically the same thing. They are not "pure indexing" but rather delivering a slice/region of data that is described by some metadata. However, even in the "pure indexing" these types of uses seem likely: observation_region[4:7, 100:120, 30:40, measure=feature.temperature, time=start:end] observation_region[4:7, 100:120, 30:40, measure=feature.pressure, time=start:end] I.e. the underlying array is 5-D, but the spatial dimensions are unnamed (and in the hypothetical there is no inherent orientation like latitude/longitude/altitude), but the 4th and 5th axes describe the variation in which instrument is make a measurement and what timestamps each measurement was made at.

3 hours, 49 minutes

Re: Suggestion: annotated exceptions

by Sascha Schlemmer

I think a clear distinction has to be made between errors that belong to the api surface of a function e.g. some `ValueError` or `ZeroDivisionError` and *exceptional* errors e,g.`KeyboardInterrupt` or `DatabaseExplodedError`. For the latter case exceptions work perfectly well as they are and it is both unhelpful and infeasible to annotate/document every exception - just let the exception bubble up and someone will handle it (might even be a process supervisor like supervisord) For the first case I think a better way to model a function, that has some *expected* failure mode is to try and write a *total* function that doesn’t use exceptions at all. Let me show what I mean: ``` def div(a: int, b: int) -> float: return a / b ``` This function tries to divide two integers, trouble is that `div` is not defined for every input we may pass it, i.e. passing `b=0` will lead to `ZeroDivisionError` being raised. In other words `div` is a *partial* function because it is not defined for every member of its domain (type of its arguments). There are two ways to amend this issue and make `div` a *total* function that is honest about its domain and co-domain (types of input arguments and return type): 1. Extending the type of the functions co-domain (return type) and making the error case explicit (see https://returns.readthedocs.io/en/latest/index.html <https://returns.readthedocs.io/en/latest/index.html> for a library that implements this style of error handling) ``` def div(a: int, b: int) -> Maybe[float]: try: return Some(a / b) except ZeroDivisionError: return Nothing() ``` or ``` def div(a: int, b: int) -> Result[float, ZeroDivisionError]: try: return Success(a / b) except ZeroDivisionError as error: return Failure(error) ``` Now `div` does return a valid instance of `Maybe` (or `Result` if more a more detailed failure case is wanted) which is either something like `Some(3.1415) ` or `Nothing` (analogous to `None` ). The caller of the function then has to deal with this and mypy will correctly warn if the user fails to do so properly e.g. ´´´ div(1,1) + 1 # mypy will give a type error ´´´ 2. Restricting the functions domain to values with defined behavior ``` def div(a: int, b: NonZeroInteger) -> float: return a / b ``` In this case the burden is put on the caller to supply valid inputs to the function and this effectively pushes the error handling/ input validation out of `div` A language that does all this really well is F# (which like python is a multi-paradigm language that offers both object-oriented and functional-programming-oriented constructs). The trouble in python at the moment is that using something like `Maybe` and `Result` is not as nice to use as exceptions due to lack of a nice syntax i.e. for exceptions we have `try`/`except` and for functional error handling we’d need pattern matching.

1 day, 12 hours

Trash bin

by Marco Sulla

That I hope it's not the place where this proposal will be sent. My idea is apparently simple: what if, anytime we create an object, instead of deleting it, we send it in a trash bin? If the object is, for some reason, recreated, we can take it from the trash bin. If there is no more memory, the trash bin will be empty. The problem is that, probably, args and kwargs used by object creation must be stored. Maybe also copied? This could slow down the object creation instead of speed it up? Could this be done by a separate thread?

1 day, 23 hours

Suggestion: annotated exceptions

by Sergio Fenoll

Hi, I was wondering if the following idea would be a useful addition to the Python language and if it could use a new PEP. I personally find myself often looking into the documentation & implementation of libraries I use to try and figure out what exceptions a function may raise. In the same vein as adding type annotations to code, I think it'd be very useful to have exception "raises" annotations, i.e. a way to annotate what exceptions a function raises. Again, like type annotations, it shouldn't be mandatory nor actually be enforced at runtime. It would purely serve as a feature that IDEs can make use of. An example of how it may look: def divide(numerator: float, denominator: float) raises [ZeroDivisionError] -> float: return numerator / denominator I'd love to know if this is an idea you'd be interested in having added to the language. Kind regards, Sergio Fenoll

2 days, 5 hours

Re: PEP 9999 (provisional): support for indexing with keyword arguments

by Eric Wieser

Thanks for the PEP, and for pinging the numpy list about it. Some comments: Sequence unpacking remains a syntax error inside subscripts: Reason: unpacking items would result it being immediately repacked into a tuple A simple counter-example is [:, *args, :], which could be treated as [(slice(None), *args, slice(None))]. When there are index objects to the left or right of *args, it enables : syntax in places that would otherwise be forbidden. I’d argue this should be folded into the scope of the PEP - if the parser for indexing is going to be overhauled, it would be nice to cross this one off. I think I requested this change independently at some point Keyword arguments must allow Ellipsis This is sort of uninteresting now, Ellipsis is not special-cased in the parser, all places where Ellipsis is allowed also allow ... - even ....__class__! a similar case occurs with setter notation It would perhaps be a good idea to recommend users declare their setter arguments as positional-only too,def __setitem__(self, index, value, /, **kwargs):, to avoid this problem Keyword-only subscripts are permitted. The positional index will be the empty tuple As discussed on the numpy mailing list, a better approach might be to not pass an argument at all, so that obj[spam=1, eggs=2] calls type(obj).__getitem__(obj, spam=1, eggs=2). Objects which want to support keyword-only indexing can provide a default of their own (and are free to choose between () and None, while objects which do not would just raise TypeError due to the missing positional argument the solution relies on the assumption that all keyword indices necessarily map into positional indices, or that they must have a name This seems like a non-issue to me. Keyword-only arguments already exist, and could be used in the relevant magic method def __getitem_ex__(self, *, x, y):. I would be inclined to remove this argument. Eric

2 days, 9 hours

Re: Suggestion: annotated exceptions

by Paul Moore

On Fri, 25 Sep 2020 at 18:28, Samuel Colvin <S(a)muelcolvin.com> wrote: >> There's also the problem that you've explicitly acknowledged, that >> exception hints are *always* going to be inaccurate, in stark contrast >> to type hints which are expected to be correct when used. > > We should draw a distinction between two kinds of "inaccurate": > * inaccuracies because data about what errors could be raised is > missing some exceptions > * errors that could happen at almost any time: > MemoryError, OverflowError, SystemError, SystemExit > > The first set of errors is not in stark contrast to type hints, > in fact they're very similar to errors in type hints which come up > from time to time. I get what you are saying, but I think you're being optimistic in assuming there's a clean distinction. It may be that you;'re thinking mostly about application code, where you are in control of what types exist in the system. But in library code, you simply cannot assume that. Consider a library that manipulates strings. There have been many attempts to write libraries that represent filesystem paths using a subclass of the string type. Suppose your library was passed one of those, and concatenated some text onto it. Possible exceptions that could in theory be raised include: * UnicodeError (if the path class checks that the value can be encoded in the filesystem encoding) * OSError (if the class enforces maximum path length or allowed character restrictions) * ValueError (if the programmer was lazy and used these rather than one of the above) as well as pretty much anything else, if the code that implemented __add__ for the path class had a bug. And of course, if libraries don't declare "expected" exceptions, application code using those libraries don't have anything to work with. Maybe you could get away with just declaring use of exception types that "you" created. But that's going to be very messy to get right. > Granted the "exceptions raised" database would be much more > immature than the type hints database to start with. But that's > not a reason not to start the error. Agreed. What *is* a reason to not start is if we can't even agree what we're building. > The second set of exceptions are a different matter, I think > in general they should not be included in the > "exceptions raised" database. If that makes exceptions > raised data invalid, then virtually all python logic in the world > that catches some exceptions but doesn't catch those > exceptions is also invalid! Precisely. So it's critical to clearly define exactly what exceptions can happen "at any time", and given that operations like indexing, basic operators, and even attribute access can call arbitrary Python code, essentially any exception can happen at any time. Unless you start mandating exception behaviour on basic operations, and then you preclude creative but unusual designs such as the path class I described above. Maybe that's not unreasonable (over-clever path classes never really became that popular) but Python has a tradition of *not* arbitrarily constraining how features can be used. > In other words: right now when we write code and catch > exceptions we generally (but not always) catch > exceptions that are likely to happen in the general > course of calling the code in the try, except block. But > not all the strange things that could conceivably happen. > > I think we should maintain that convention but provide a > way to remind developers when they forget to catch an exception. How do you distinguish between a developer "forgetting" to catch an exception and deliberately letting it bubble up. I'm a strong -1 on any proposal that makes people explicitly say "I meant to let this through" (it's essentially leading to checked exceptions, as well as violating the principle I stated above about not constraining how people use features - in this case the feature of exceptions bubbling up). And it's obviously impossible to make people declare when they forgot to catch an exception. > I guess overall, the question is: would it be useful to have a > way to check what exceptions a block of code might raise? > > Yes, massively. You state that without any justification, and it's the key point. My view is that it's (barely) possible that it might be occasionally convenient, but no-one has really demonstrated a benefit that I care about at this point. And to be honest, claims like "tools might be able to warn about certain types of error" are too vague to carry much weight. I've been developing Python for many years, and work on pip, which is a major codebase that's full of legacy weirdness - and I don't think I've *ever* hit an issue where I've felt that having a checker warn me that I haven't caught a type of exception would be useful. (Many, many times I've had to fix issues where uncaught exceptions were raised, but they have basically always been logic errors where fixing the underlying issue is the correct action, not catching the exception. And that raises another point - often, uncaught exceptions are *better* than carefully catching everything. Better for the developer, not for the end user, but I'll come to that. If an uncaught exception happens, there's a full traceback that is often invaluable in debugging what went wrong. Certainly, a traceback is the worst sort of thing for the user to see, but I've debugged way too many systems where some bright spark has added an exception handler that converts exceptions to "user friendly" messages, and in doing so completely discards all of the context that I need to fix the issue. One potential risk with this proposal is that it will encourage a culture of thoughtlessly catching exceptions and "handling" them, often in a way that makes maintenance and support orders of magnitude harder. Consider the following: def some_function() -> None[raises AppError]: try: do() some() complicated() piece() of() work() except Exception as e: log.debug("Caught exception " + str(e)) raise AppError("Unexpected problem with some_function()") That looks like it's doing a good job of exception handling, but it would be a nightmare to debug. It hides the underlying error in a debug message (that the user might well not have enabled). It doesn't retain the call stack. It puts a huge chunk of code in a try...except block. Clearly it can be improved, and ideally someone with experience would point out the issues in code review. But in the real world, a junior developer, armed with a style guide that says "functions must declare the exceptions they could raise, and must log any unexpected problems" could have this in production before you could say "real life's never as ideal as you'd hope" :-) > Would it be a lot of work? yes, massive. > > Would the work be worth the reward? Hard to say right now. Unless someone defines the reward more explicitly, it's easy - no. Give me £100 and I might give you something in return. That's the proposition here at the moment... Sorry - this comes across as rather negative. Your arguments are well-stated and I appreciate the time you're taking to address the points I'm making. It's just that I remain unconvinced, I'm afraid. Paul

3 days

Suggest mentioning venv --upgrade in installer output & release notes

by Steve Barnes

If you are upgrading a maintenance version (following the major.minor.maintenance naming) of python, e.g. 3.8.5 -> 3.8.6 it is also a good idea to update any 3.8 venvs that you may be using. The venv –upgrade flag is reasonably new and I think that relatively few people will have heard about it. I was wondering if it might not be a good idea to issue a reminder, at the end of installation &/or mention reasonably prominently in the release notes, that it is a good idea to upgrade any venvs of the same major.minor version – especially as symlinked venvs will have been upgraded but copy ones will not have been which can lead to inconsistent behaviour. Steve (Gadget) Barnes

3 days, 13 hours

Jump to page:

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas