Mailman 3 July 2017 - Python-ideas

tempfile.TemporaryDirectory() should be able to create temporary directory at a given arbitrary place
by George Fischhof 18 Jul '17

18 Jul '17

Hi there, I used tempfile.TemporaryDirectory(). On first usage it was good, but on second one there was a need to create tempopray directory and files in it a given place. (It needed for a test). And I found that TemporaryDirectory() is not able to do this. So my idea is to implement this behaviour with an addittional path parameter in it. BR, George

3 3

Is this PEP viable?
by Evan Adler 17 Jul '17

17 Jul '17

I would like to submit the following proposal. In the logging module, I would like handlers (like file handlers and stream handlers) to have a field for exc_info printing. This way, a call to logger.exception() will write the stack trace to the handlers with this flag set, and only print the message and other info to handlers without the flag set. This allows a single logger to write to a less detailed console output, a less detailed run log, and a more detailed error log.

2 1

Arguments to exceptions
by Ken Kundert 17 Jul '17

17 Jul '17

All, Here is a proposal for enhancing the way BaseException handles arguments. -Ken Rationale ========= Currently, the base exception class takes all the unnamed arguments passed to an exception and saves them into args. In this way they are available to the exception handler. A commonly used and very useful idiom is to extract an error message from the exception by casting the exception to a string. If you do so while passing one argument, you get that argument as a string: >>> try: ... raise Exception('Hey there!') ... except Exception as e: ... print(str(e)) Hey there! However, if more than one argument is passed, you get the string representation of the tuple containing all the arguments: >>> try: ... raise Exception('Hey there!', 'Something went wrong.') ... except Exception as e: ... print(str(e)) ('Hey there!', 'Something went wrong.') That behavior does not seem very useful, and I believe it leads to people passing only one argument to their exceptions. An example of that is the system NameError exception: >>> try: ... foo ... except Exception as e: ... print('str:', str(e)) ... print('args:', e.args) str: name 'foo' is not defined args: ("name 'foo' is not defined",) Notice that the only argument is the error message. If you want access to the problematic name, you have to dig it out of the message. For example ... >>> import Food >>> try: ... import meals ... except NameError as e: ... name = str(e).split("'")[1] # <-- fragile code ... from difflib import get_close_matches ... candidates = ', '.join(get_close_matches(name, Food.foods, 1, 0.6)) ... print(f'{name}: not found. Did you mean {candidates}?') In this case, the missing name was needed but not directly available. Instead, the name must be extracted from the message, which is innately fragile. The same is true with AttributeError: the only argument is a message and the name of the attribute, if needed, must be extracted from the message. Oddly, with a KeyError it is the opposite situation, the name of the key is the argument and no message is included. With IndexError there is a message but no index. However none of these behaviors can be counted on; they could be changed at any time. When writing exception handlers it is often useful to have both a generic error message and access to the components of the message if for no other reason than to be able to construct a better error message. However, I believe that the way the arguments are converted to a string when there are multiple arguments discourages this. When reporting an exception, you must either give one argument or add a custom __str__ method to the exception. To do otherwise means that the exception handlers that catch your exception will not have a reasonable error message and so would be forced to construct one from the arguments. This is spelled out in PEP 352, which explicitly recommends that there be only one argument and that it be a helpful human readable message. Further it suggests that if more than one argument is required that Exception should be subclassed and the extra arguments should be attached as attributes. However, the extra effort required means that in many cases people just pass an error message alone. This approach is in effect discouraging people from adding additional arguments to exceptions, with the result being that if they are needed by the handler they have to be extracted from the message. It is important to remember that the person that writes the exception handler often does not raise the exception, and so they just must live with what is available. As such, a convention that encourages the person raising the exception to include all the individual components of the message should be preferred. That is the background. Here is my suggestion on how to improve this situation. Proposal ======== I propose that the Exception class be modified to allow passing a message template as a named argument that is nothing more than a format string that interpolates the exception arguments into an error message. If the template is not given, the arguments would simply be converted to strings individually and combined as in the print function. So, I am suggesting the BaseException class look something like this: class BaseException: def __init__(self, *args, **kwargs): self.args = args self.kwargs = kwargs def __str__(self): template = self.kwargs.get('template') if template is None: sep = self.kwargs.get('sep', ' ') return sep.join(str(a) for a in self.args) else: return template.format(*self.args, **self.kwargs) Benefits ======== Now, NameError could be defined as: class NameError(Exception): pass A NameError would be raised with: try: raise NameError(name, template="name '{0}' is not defined.") except NameError as e: name = e.args[0] msg = str(e) ... Or, perhaps like this: try: raise NameError(name=name, template="name '{name}' is not defined.") except NameError as e: name = e.kwargs['name'] msg = str(e) ... One of the nice benefits of this approach is that the message printed can be easily changed after the exception is caught. For example, it could be converted to German. try: raise NameError(name, template="name '{0}' is not defined.") except NameError as e: print('{}: nicht gefunden.'.format(e.args[0])) A method could be provided to generate the error message from a custom format string: try: raise NameError(name, template="name '{0}' is not defined.") except NameError as e: print(e.render('{}: nicht gefunden.')) Another nice benefit of this approach is that both named and unnamed arguments to the exception are retained and can be processed by the exception handler. Currently this is only true of unnamed arguments. And since named arguments are not currently allowed, this proposal is completely backward compatible. Of course, with this change, the built-in exceptions should be changed to use this new approach. Hopefully over time, others will change the way they write exceptions to follow suit, making it easier to write more capable exception handlers. Conclusion ========== Sorry for the length of the proposal, but I thought it was important to give a clear rationale for it. Hopefully me breaking it into sections made it easier to scan. Comments?

19 58

Custom Code Folding: Standardized Rules and Syntax?
by Connor Farrell 17 Jul '17

17 Jul '17

Background: I work in scientific computing and use Community Pycharm IDE. I'm a religious follower of the 'readability counts' mantra, and two things I find myself doing often are: - Writing custom code folds to segregate code, from groups of classes in a file, to groups of lines in an individual function. While spacing works great to separate ideas, my IDE allows me to collapse the entirety of the code in exchange for a line of English. For my purposes, this enhances readability immensely, as first time users are confronted with an explanation of the contents, rather than the code itself with a comment on top. I find comments don't draw the eye, and also don't have the ability to their code as well. - Writing high level code, such as __init__ calls for large aggregates, with one keyworded argument per line (plus dict unpackings at the end), sort of like a simple XML file. This allows me to make parameters explicit for other users, and optionally provide a comment indicating physical units, cite sources, and/or give a list of tag/enum options for every parameter. In the end I have 30+ line inits, but the readability is 10x greater. My IDE doesn't yet offer to fold long parameter lists by default, but I think it makes sense. In the end, I end up with very well folded code (except for large parameter lists) and a bunch of co-workers asking about all the "editor-fold" comments that don't work in their API. Python was a trail-blazer in terms of emphasizing the importance of code readability and effective syntax. I think that we should consider some sort of standard for folding comments, if not only to promote stronger code organizations. I know standards don't usually interact with IDEs, but hey, the 'typing' module is pretty dang nice. TL;DR: Code folding is great, custom code folding is great, let's upgrade it to a language feature. Cheers

4 3

Re: [Python-ideas] Python-ideas Digest, Vol 128, Issue 41
by Connor Farrell 16 Jul '17

16 Jul '17

Thanks for your feedback, I guess I was a little unclear. In short, I was thinking of a pair of comment tokens (something like #<<, #>>, idk) that would indicate a code fold, like what virtually all IDEs do for classes and methods, but with more granularity. It would allow devs to better organize their code. I agree with what you're saying about separation of language and text editor, but we already have the *typing* module in 3.6, so improved linting and communication is apparently game. I have no desire to get the interpreter involved, this is pure linter. A good example would be something like: class A: #<< INTERFACE def foo(): .... def bar(): .... #>> #<< BACKEND def _guts(): .... #>> Would become something like: class A: *>+ INTERFACE* * >+ BACKEND* Where *modern* editors should fold the code. It provides an optional additional layer of granularity above the current system, but has no effect on behaviour otherwise. It increases visual information density in complex functions, large classes, and lets you group classes. It looks stupid with only three functions, but at larger sizes, or with complex code, I'd rather see: def func(f: Callable[[float],float, float]) -> None: *>+* *Input validation* *>+ Some complex algorithm* * >+ Some other complex algorithm* * >+ Generating plot* It adds a human explanation of the code within, an additional level(s) of organization, and easier navigation. It's not for everyone, but I feel like it improves on the idea of modular code at for active devs, without any drawbacks for library users. On Sun, Jul 16, 2017 at 12:00 PM, <python-ideas-request(a)python.org> wrote: > Send Python-ideas mailing list submissions to > python-ideas(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/python-ideas > or, via email, send a message with subject or body 'help' to > python-ideas-request(a)python.org > > You can reach the person managing the list at > python-ideas-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Python-ideas digest..." > > > Today's Topics: > > 1. Re: Custom Code Folding: Standardized Rules and Syntax? > (Steven D'Aprano) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 17 Jul 2017 01:15:59 +1000 > From: Steven D'Aprano <steve(a)pearwood.info> > To: python-ideas(a)python.org > Subject: Re: [Python-ideas] Custom Code Folding: Standardized Rules > and Syntax? > Message-ID: <20170716151559.GW3149(a)ando.pearwood.info> > Content-Type: text/plain; charset=us-ascii > > Hi Connor, and welcome! > > On Sun, Jul 16, 2017 at 10:37:26AM -0400, Connor Farrell wrote: > > Background: I work in scientific computing and use Community Pycharm IDE. > > > > I'm a religious follower of the 'readability counts' mantra, and two > things > > I find myself doing often are: > > - Writing custom code folds to segregate code, from groups of classes in > a > > file, to groups of lines in an individual function. While spacing works > > great to separate ideas, my IDE allows me to collapse the entirety of the > > code in exchange for a line of English. For my purposes, this enhances > > readability immensely, as first time users are confronted with an > > explanation of the contents, rather than the code itself with a comment > on > > top. I find comments don't draw the eye, and also don't have the ability > to > > their code as well. > > I'm afraid I'm having a lot of difficulty understanding this. I think > the last sentence is missing a word. Comments don't have the ability to > **what** their (whose?) code? > > Which IDE are you using? When you say it collapses the "entirety of the > code", do you mean the entire file? > > > > - Writing high level code, such as __init__ calls for large aggregates, > > with one keyworded argument per line (plus dict unpackings at the end), > > sort of like a simple XML file. > > Even if I accept that this is a reasonable design for __init__, I would > not agree that it is a reasonable design for "high level code" in > general. > > > > This allows me to make parameters explicit > > for other users, and optionally provide a comment indicating physical > > units, cite sources, and/or give a list of tag/enum options for every > > parameter. In the end I have 30+ line inits, but the readability is 10x > > greater. > > Perhaps I might be convinced if I saw some actual code, but from your > description alone, it doesn't sound particularly more readable. Why > would I want to read citations in the parameter list of a method? I want > to call the method, not do peer review on the theory behind it. > > > > My IDE doesn't yet offer to fold long parameter lists by default, > > but I think it makes sense. > > *shrug* > > Personally, I don't find code folding a big help. Perhaps once in a blue > moon. I'm glad you like it and that it helps you. > > > > In the end, I end up with very well folded code (except for large > parameter > > lists) and a bunch of co-workers asking about all the "editor-fold" > > comments that don't work in their API. > > I'm afraid I'm not understanding you here either. What's an > "editor-fold" comment? What do they mean by API? API for which > application? How does the programming interface to an application relate > to code folding in a text editor? > > > > Python was a trail-blazer in terms of emphasizing the importance of code > > readability and effective syntax. I think that we should consider some > sort > > of standard for folding comments, if not only to promote stronger code > > organizations. I know standards don't usually interact with IDEs, but > hey, > > the 'typing' module is pretty dang nice. > > > > TL;DR: Code folding is great, custom code folding is great, let's upgrade > > it to a language feature. > > What does that even mean? Are you suggesting that the Python interpreter > should raise a SyntaxError or something if your code was written in an > editor that didn't support code folding? How would it know? > > Python is a programming language. The source code is text. I should be > able to write Python code in NotePad if I want. Why should the Python > core devs try to force every text editor and IDE fold code exactly the > same way? That sounds awful to me. People choose different editors > because they like different features, and that may include the > particular way the editor folds code. Or to not fold it at all. > > I'm sorry to be so negative, but I don't understand your proposal, and > the bits that I *think* I understand sound pretty awful to me. Perhaps > you can explain a bit better what you mean and why it should be a > language feature, apart from "I want everyone to lay out their source > code the way I do". Because that's what it sounds like to me. > > > -- > Steve > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > https://mail.python.org/mailman/listinfo/python-ideas > > > ------------------------------ > > End of Python-ideas Digest, Vol 128, Issue 41 > ********************************************* >

3 2

PEP: Hide implementation details in the C API
by Victor Stinner 14 Jul '17

14 Jul '17

Hi, This is the first draft of a big (?) project to prepare CPython to be able to "modernize" its implementation. Proposed changes should allow to make CPython more efficient in the future. The optimizations themself are out of the scope of the PEP, but some examples are listed to explain why these changes are needed. For the background, see also my talk at the previous Python Language Summit at Pycon US, Portland OR: "Keeping Python competitive" https://lwn.net/Articles/723752/#723949 "Python performance", slides (PDF): https://github.com/haypo/conf/raw/master/2017-PyconUS/summit.pdf Since this is really the first draft, I didn't assign a PEP number to it yet. I prefer to wait for a first feedback round. Victor PEP: xxx Title: Hide implementation details in the C API Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner(a)gmail.com>, Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 31-May-2017 Abstract ======== Modify the C API to remove implementation details. Add an opt-in option to compile C extensions to get the old full API with implementation details. The modified C API allows to more easily experiment new optimizations: * Indirect Reference Counting * Remove Reference Counting, New Garbage Collector * Remove the GIL * Tagged pointers Reference counting may be emulated in a future implementation for backward compatibility. Rationale ========= History of CPython forks ------------------------ Last 10 years, CPython was forked multiple times to attempt different CPython enhancements: * Unladen Swallow: add a JIT compiler based on LLVM * Pyston: add a JIT compiler based on LLVM (CPython 2.7 fork) * Pyjion: add a JIT compiler based on Microsoft CLR * Gilectomy: remove the Global Interpreter Lock nicknamed "GIL" * etc. Sadly, none is this project has been merged back into CPython. Unladen Swallow looses its funding from Google, Pyston looses its funding from Dropbox, Pyjion is developed in the limited spare time of two Microsoft employees. One hard technically issue which blocked these projects to really unleash their power is the C API of CPython. Many old technical choices of CPython are hardcoded in this API: * reference counting * garbage collector * C structures like PyObject which contains headers for reference counting and the garbage collector * specific memory allocators * etc. PyPy ---- PyPy uses more efficient structures and use a more efficient garbage collector without reference counting. Thanks to that (but also many other optimizations), PyPy succeeded to run Python code up to 5x faster than CPython. Plan made of multiple small steps ================================= Step 1: split Include/ into subdirectories ------------------------------------------ Split the ``Include/`` directory of CPython: * ``python`` API: ``Include/Python.h`` remains the default C API * ``core`` API: ``Include/core/Python.h`` is a new C API designed for building Python * ``stable`` API: ``Include/stable/Python.h`` is the stable ABI Expect declarations to be duplicated on purpose: ``#include`` should be not used to include files from a different API to prevent mistakes. In the past, too many functions were exposed *by mistake*, especially symbols exported to the stable ABI by mistake. At this point, ``Include/Python.h`` is not changed at all: zero risk of backward incompatibility. The ``core`` API is the most complete API exposing *all* implementation details and use macros for best performances. XXX should we abandon the stable ABI? Never really used by anyone. Step 2: Add an opt-in API option to tools building packages ----------------------------------------------------------- Modify Python packaging tools (distutils, setuptools, flit, pip, etc.) to add an opt-in option to choose the API: ``python``, ``core`` or ``stable``. For example, debuggers like ``vmprof`` need need the ``core`` API to get a full access to implementation details. XXX handle backward compatibility for packaging tools. Step 3: first pass of implementation detail removal --------------------------------------------------- Modify the ``python`` API: * Add a new ``API`` subdirectory in the Python source code which will "implement" the Python C API * Replace macros with functions. The implementation of new functions will be written in the ``API/`` directory. For example, Py_INCREF() becomes the function ``void Py_INCREF(PyObject *op)`` and its implementation will be written in the ``API`` directory. * Slowly remove more and more implementation details from this API. Modifications of these API should be driven by tests of popular third party packages like: * Django with database drivers * numpy * scipy * Pillow * lxml * etc. Compilation errors on these extensions are expected. This step should help to draw a line for the backward incompatible change. Goal: remove a few implementation details but don't break numpy and lxml. Step 4 ------ Switch the default API to the new restricted ``python`` API. Help third party projects to patch their code: don't break the "Python world". Step 5 ------ Continue Step 3: remove even more implementation details. Long-term goal to complete this PEP: Remove *all* implementation details, remove all structures and macros. Alternative: keep core as the default API ========================================= A smoother transition would be to not touch the existing API but work on a new API which would only be used as an opt-in option. Similar plan used by Gilectomy: opt-in option to get best performances. It would be at least two Python binaries per Python version: default compatible version, and a new faster but incompatible version. Idea: implementation of the C API supporting old Python versions? ================================================================= Open questions. Q: Would it be possible to design an external library which would work on Python 2.7, Python 3.4-3.6, and the future Python 3.7? Q: Should such library be linked to libpythonX.Y? Or even to a pythonX.Y binary which wasn't built with shared library? Q: Would it be easy to use it? How would it be downloaded and installed to build extensions? Collaboration with PyPy, IronPython, Jython and MicroPython =========================================================== XXX to be done Enhancements becoming possible thanks to a new C API ==================================================== Indirect Reference Counting --------------------------- * Replace ``Py_ssize_t ob_refcnt;`` (integer) with ``Py_ssize_t *ob_refcnt;`` (pointer to an integer). * Same change for GC headers? * Store all reference counters in a separated memory block (or maybe multiple memory blocks) Expected advantage: smaller memory footprint when using fork() on UNIX which is implemented with Copy-On-Write on physical memory pages. See also `Dismissing Python Garbage Collection at Instagram <https://engineering.instagram.com/dismissing-python-garbage-collection-at-i…>`_. Remove Reference Counting, New Garbage Collector ------------------------------------------------ If the new C API hides well all implementation details, it becomes possible to change fundamental features like how CPython tracks the lifetime of an object. * Remove ``Py_ssize_t ob_refcnt;`` from the PyObject structure * Replace the current XXX garbage collector with a new tracing garbage collector * Use new macros to define a variable storing an object and to set the value of an object * Reimplement Py_INCREF() and Py_DECREF() on top of that using a hash table: object => reference counter. XXX PyPy is only partially successful on that project, cpyext remains very slow. XXX Would it require an opt-in option to really limit backward compatibility? Remove the GIL -------------- * Don't remove the GIL, but replace the GIL with smaller locks * Builtin mutable types: list, set, dict * Modules * Classes * etc. Backward compatibility: * Keep the GIL Tagged pointers --------------- https://en.wikipedia.org/wiki/Tagged_pointer Common optimization, especially used for "small integers". Current C API doesn't allow to implement tagged pointers. Tagged pointers are used in MicroPython to reduce the memory footprint. Note: ARM64 was recently extended its address space to 48 bits, causing issue in LuaJIT: `47 bit address space restriction on ARM64 <https://github.com/LuaJIT/LuaJIT/issues/49>`_. Misc ideas ---------- * Software Transactional Memory? See `PyPy STM <http://doc.pypy.org/en/latest/stm.html>`_ Idea: Multiple Python binaries ============================== Instead of a single ``python3.7``, providing two or more binaries, as PyPy does, would allow to experiment more easily changes without breaking the backward compatibility. For example, ``python3.7`` would remain the default binary with reference counting and the current garbage collector, whereas ``fastpython3.7`` would not use reference counting and a new garbage collector. It would allow to more quickly "break the backward compatibility" and make it even more explicit than only prepared C extensions will be compatible with the new ``fastpython3.7``. cffi ==== XXX Long term goal: "more cffi, less libpython". Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

9 20

JavaScript-Style Object Creation in Python (using a constructor function instead of a class to create objects)
by Simon Ramstedt 13 Jul '17

13 Jul '17

Hi, do you have an opinion on the following? Wouldn't it be nice to define classes via a simple constructor function (as below) instead of a conventional class definition? *conventional*: class MyClass(ParentClass): def __init__(x): self._x = x def my_method(y): z = self._x + y return z *proposed*: def MyClass(x): self = ParentClass() def my_method(y): z = x + y return z self.my_method = my_method # that's cumbersome (see comments below) return self Here are the pros and cons I could come up with for the proposed method: (+) Simpler and more explicit. (+) No need to create attributes (like `self._x`) just to pass something from `__init__` to another method. (+) Default arguments / annotations for methods could be different for each class instance. Adaptive defaults wouldn't have to simulated with a None. (+) Class/instance level imports would work. (-/+) Speed: The `def`-based objects take 0.6 μs to create while the `class`-based objects take only 0.4 μs. For method execution however the closure takes only 0.15 μs while the proper method takes 0.22 μs (script <https://gist.github.com/rmst/78b2b0f56a3d9ec13b1ec6f3bd50aa9c>). (-/+) Checking types: In the proposed example above the returned object wouldn't know that it has been created by `MyClass`. There are a couple of solutions to that, though. The easiest to implement would be to change the first line to `self = subclass(ParentClass())` where the subclass function looks at the next item in the call stack (i.e. `MyClass`) and makes it the type of the object. Another solution would be to have a special rule for functions with capital first letter returning a single object to append itself to the list of types of the returned object. Alternatively there could be a special keyword e.g. `classdef` that would be used instead of `def` if we wouldn't want to rely on the name. (-) The current syntax for adding a function to an object is cumbersome. That's what is preventing me from actually using the proposed pattern. But is this really the only reason for not using it? And if so, wouldn't that be a good argument for enabling something like below? *attribute function definitions*: def MyClass(x): self = ParentClass() def self.my_method(y): z = x + y return z return self or alternatively *multiline lambdas*: def MyClass(x): self = ParentClass() self.my_method = (y): z = x + y return z return self Cheers, Simon

25 80

Improving Catching Exceptions
by Sven R. Kunze 08 Jul '17

08 Jul '17

Hi folks, just one note I'd like to dump here. We usually teach our newbies to catch exceptions as narrowly as possible, i.e. MyModel.DoesNotExist instead of a plain Exception. This works out quite well for now but the number of examples continue to grow where it's not enough. There are at least three examples I can name off the top of my head: 1) nested StopIteration - PEP 479 2) nested ImportError 3) nested AttributeError 1) is clear. 2) usually can be dealt with by applying the following pattern: try: import user except ImportError: import sys if sys.exc_info()[2].tb_next: raise Chris showed how to deal with 3). Catching nested exception is not what people want many times. Am I the only one getting the impression that there's a common theme here? Regards, Sven

13 41

Consider allowing the use of abstractmethod without metaclasses
by Neil Girdhar 08 Jul '17

08 Jul '17

I want to use abstractmethod, but I have my own metaclasses and I don't want to build composite metaclasses using abc.ABCMeta. Thanks to PEP 487, one approach is to facator out the abstractmethod checks from ABCMeta into a regular (non-meta) class. So, my first suggestion is to split abc.ABC into two pieces, a parent regular class with metaclass "type": class AbstractBaseClass: def __init_subclass__(cls): # Compute set of abstract method names abstracts = {name for name, value in vars(cls).items() if getattr(value, "__isabstractmethod__", False)} for base in cls.__bases__: for name in getattr(base, "__abstractmethods__", set()): value = getattr(cls, name, None) if getattr(value, "__isabstractmethod__", False): abstracts.add(name) cls.__abstractmethods__ = frozenset(abstracts) My alternative suggestion is to move this logic directly into "type" so that all classes have this logic, and then move abstractmethod into builtins. Of course, this isn't pressing since I can do this in my own code, it's just a suggestion from a neatness standpoint. Best, Neil

1 0

Re: [Python-ideas] Arguments to exceptions
by Jeff Walker 06 Jul '17

06 Jul '17

Paul, I think you are fixating too much on Ken's example. I think I understand what he is saying and I agree with him. It is a problem I struggle with routinely. It occurs in the following situations: 1. You are handling an exception that you are not raising. This could be because Python itself is raising the exception, as in Ken's example, or it could be raised by some package you did not write. 2. You need to process or transform the message in some way. Consider this example: import json >>> s = '{"abc": 0, "cdf: 1}' >>> try: ... d = json.loads(s) ... except Exception as e: ... print(e) ... print(e.args) Unterminated string starting at: line 1 column 12 (char 11) ('Unterminated string starting at: line 1 column 12 (char 11)',) Okay, I have caught an exception for which I have no control over how the exception was raised. Now, imagine that I am writing an application that highlights json errors in place. To do so, I would need the line and column numbers to highlight the location of the error, and ideally I'd like to strip them from the base message and just show that. You can see from my second print statement that the line and column numbers were not passed as separate arguments. Thus I need to parse the error message to extract them. Not a difficult job, but fragile. Any change to the error message could break my code. I don't know what this code smell is that people keep referring to, but to me, that code would smell. Jeff > On 3 July 2017 at 09:59, Ken Kundert <python-ideas(a)shalmirane.com> wrote: > > I think in trying to illustrate the existing behavior I made things more > > confusing than they needed to be. Let me try again. > > > > Consider this code. > > > > >>> import Food > > >>> try: > > ... import meals > > ... except NameError as e: > > ... name = str(e).split("'")[1] # <-- fragile code > > ... from difflib import get_close_matches > > ... candidates = ', '.join(get_close_matches(name, Food.foods, 1, 0.6)) > > ... print(f'{name}: not found. Did you mean {candidates}?') > > > > In this case *meals* instantiates a collection of foods. It is a Python file, > > but it is also a data file (in this case the user knows Python, so Python is > > a convenient data format). In that file thousands of foods may be instantiated. > > If the user misspells a food, I would like to present the available > > alternatives. To do so, I need the misspelled name. The only way I can get it > > is by parsing the error message. > > As Steven pointed out, this is a pretty good example of a code smell. > My feeling is that you may have just proved that Python isn't quite as > good a fit for your data file format as you thought - or that your > design has flaws. Suppose your user had a breakfast menu, and did > something like: > > if now < lunchtim: # Should have been "lunchtime" > > Your error handling will be fairly confusing in that case. > > > That is the problem. To write the error handler, I need the misspelled name. > > The only way to get it is to extract it from the error message. The need to > > unpack information that was just packed suggests that the packing was done too > > early. That is my point. > > I don't have any problem with *having* the misspelled name as an > attribute to the error, I just don't think it's going to be as useful > as you hope, and it may indeed (as above) encourage people to use it > without thinking about whether there might be problems with using > error handling that way. > > > Fundamentally, pulling the name out of an error message is a really bad coding > > practice because it is fragile. The code will likely break if the formatting or > > the wording of the message changes. But given the way the exception was > > implemented, I am forced to choose between two unpleasant choices: pulling the > > name from the error message or not giving the enhanced message at all. > > Or using a different approach. ("Among our different approaches...!" > :-)) Agreed that's also an unpleasant choice at this point. > > > What I am hoping to do with this proposal is to get the Python developer > > community to see that: > > 1. The code that handles the exception benefits from having access to the > > components of the error message. In the least it can present the message to > > the user is the best possible way. Perhaps that means enforcing a particular > > style, or presenting it in the user's native language, or perhaps it means > > providing additional related information as in the example above. > > I see it as a minor bug magnet, but not really a problem in principle. > > > 2. The current approach to exceptions follows the opposite philosophy, > > suggesting that the best place to construct the error message is at the > > source of the error. It inadvertently puts obstacles in place that make it > > difficult to customize the message in the handler. > > It's more about implicitly enforcing the policy of "catch errors over > as small a section of code as practical". In your example, you're > trapping NameError from anywhere in a "many thousands" of line file. > That's about as far from the typical use of one or two lines in a try > block as you can get. > > > 3. Changing the approach in the BaseException class to provide the best of both > > approaches provides considerable value and is both trivial and backward > > compatible. > > A small amount of value in a case we don't particularly want to encourage. > Whether it's trivial comes down to implementation - I'll leave that to > whoever writes the PR to demonstrate. (Although if it *is* trivial, is > it something you could write a PR for?) > > Also, given that this would be Python 3.7 only, would people needing > this functionality (only you have expressed a need so far) be OK with > either insisting their users go straight to Python 3.7, or including > backward compatible code for older versions? > > Overall, I'm -0 on this request (assuming it is trivial to implement - > I certainly don't feel it's worth significant implementation effort). > > Paul

8 11