Mailman 3 August 2018 - Python-ideas

Python certification
by Patrick Morris 10 Aug '18

10 Aug '18

Hi all I know this might be a contentious question but ... Does anyone know of any PSF approved training or certifications for Python developers? There is this 10 year old page https://wiki.python.org/psf/Certification%20Proposal but I dont see any evidence that this was ever progressed past the discussion of it. Assuming there is no such thing as PSF approval, are there any certifications / training course that people would recommend, or would recognise for someone seeking employment with them? thanks Patrick [cross posted from python-uk list]

6 8

Revisiting dedicated overloadable boolean operators
by Todd 09 Aug '18

09 Aug '18

Coming back to the previous discussion about a new set of overloadable boolean operators [1], I have an idea for overloadable boolean operators that I think might work. The idea would be to define four new operators that take two inputs and return a boolean result based on them. This behavior can be overridden in appropriate dunder methods. These operators would have similar precedence to existing logical operators. The operators would be: bNOT - boolean "not" bAND - boolean "and" bOR - boolean "or" bXOR - boolean "xor" With corresponding dunder methods: __bNOT__ and _rbNOT__ (or __r_bNOT__) __bAND__ and _rbAND__ (or __r_bAND__) __bOR__ and _rbOR__ (or __r_bOR__) __bXOR__ and _rbXOR__ (or __r_bXOR__) The basic idea is that the "b" is short for "boolean", and we change the rest of the operator to upercase to avoid confusions with the existing operators. I think these operators would be preferably to the proposals so far (see [1] again) for a few reasons: 1. They are not easy to mistake with existing operators. They are clearly not similar to the existing bitwise operators like & or |, and although they are clearly related to the "not", "and", and "or" I think they are distinct enough that it should not be easy to confuse the two or accidentally use one in place of the other. 2. They are related to the operations they carry out, which is also an advantage over the existing bitwise operators. 3. The corresponding dunder methods (such as __bAND__ and _rbAND__) are obvious and not easily confused with anything else. 4. The unusual capitalization means they are not likely to be used much in existing Python code. It doesn't fall under any standard capitalization scheme I am aware of. 5. At least for english the capitalization means they are not easy to confuse with existing words. For example Band is a word, but it is not likely to be capitalized as bAND. As to why this is useful, the overall problem is that the current logical operators, like and, or, and not, cannot be overloaded, which means projects like numpy and SQLAlchemy instead have to (ab)use bitwise operators to define their own boolean operations (for example elementwise "and" in numpy arrays). This has a variety of problems, such not having appropriate precedence leading to precedence errors being common, and the simple fact that this precludes them from using the bitwise operators for bitwise operations. There was a proposal to allow overloading boolean operators in Pep-335 [2], but that PEP was rejected for a variety of very good reasons. I think none of those reasons (besides the conversation fizzling out) apply to my proposal. So the alternative proposal that has been floating around is to instead define new operators specifically for this. Although there seemed to be some support for this in principle, the actually operators so far have not met with much enthusiasm. So far the main operators proposed so far seem to be: 1. Double bitwise operators, such as && and ||. These have the disadvantage of looking like they should be a type of bitwise operator. 2. the existing operators, with some non-letter character at the front and back, like ".and.". These have the advantage that they are currently not valid syntax in most cases, but I think are too similar to existing logical operators, to easy to confuse, and it is not immediately obvious in what way they should differ from existing operators. They also mean different things in other languages. So I think my proposal addresses the main issues raised with existing proposals, but has the downside that it requires new keywords. Thoughts? [1] https://mail.python.org/pipermail/python-ideas/2015-November/037207.html [2] https://www.python.org/dev/peps/pep-0335/

14 32

File format for automatic and manual tests
by Victor Porton 09 Aug '18

09 Aug '18

This is an idea of a new PEP. I propose to create a portable file format which will list command line options to run Python scripts with dual purpose: 1. for automatic tests executing scripts from this file (and optionally checking their stdout against specified values). 2. for running the scripts in a Python debugger. The scripts may be either real command line applications written in Python or scripts created specifically for testing. Currently I use PyCharm to debug my command line application. PyCharm allows to define and store different command line options for Python scripts. This has two drawbacks: 1. It is not portable and this makes me not to submit the PyCharm configuration into my GitHub repo. 2. I can't (at least easily) use these command line profiles for automated testing. I propose to create the file format listing such test/debug cases based on options available in PyCharm, that is: - Python script path - command line parameters - environment variables - Python interpreter and interpreter options - working directory - tweaks of PYTHONPATH to include our source directory as necessary - redirect input from It would be nice to add "variables" (substituted with strings) to our file format. Automated tests should support iterations of command lines with different vars substituted. I hope a future version of PyCharm will support our file format.

9 10

Make "yield" inside a with statement a SyntaxError
by Ken Hilton 08 Aug '18

08 Aug '18

This mostly springs off of a comment I saw in some thread. The point of a with statement is that it ensures that some resource will be disposed of, yes? For example, this: with open(filename) as f: contents = f.read() is better than this: contents = open(filename).read() because the former definitely closes the file while the latter relies on garbage collection? The point of a yield expression is to suspend execution. This is nice for efficient looping because instead of having to hold all results in memory, each result can be consumed immediately, yes? Therefore this: def five_to_one(): for i in range(4): yield 5 - i is better than this: def five_to_one(): result = [] for i in range(4): result.append(5 - i) return result because the former suspends execution of "five_to_one" while the latter holds all five results in memory? Now, let's take a look at the following scenario: def read_multiple(*filenames): for filename in filenames: with open(filename) as f: yield f.read() Can you spot the problem? The "with open(filename)" statement is supposed to ensure that the file object is disposed of properly. However, the "yield f.read()" statement suspends execution within the with block, so if this happened: for contents in read_multiple('chunk1', 'chunk2', 'chunk3'): if contents == 'hello': break and the contents of "chunk2" were "hello" then the loop would exit, and "chunk2" would never be closed! Yielding inside a with block, therefore, doesn't make sense and can only lead to obscure bugs. The proper way to define the "read_multiple" function would be like so: def read_multiple(*filenames): for filename in filenames: with open(filename) as f: contents = f.read() yield contents Save the contents in a variable somewhere, then yield the variable, instead of suspending execution within a context manager. I believe all possible cases where one would yield inside a context manager can be covered by saving anything required from the context manager and then yielding the results outside. Therefore, I propose making a "yield" inside a with block become a SyntaxError. This means the first "read_multiple" definition I presented will become illegal and fail *at compile-time*. However, it is still legal to define a generator inside a with block: def pass_file_chars(oldfunc): with open('secretfile') as f: contents = f.read() @functools.wraps def newfunc(*args, **kwargs): for char in contents: yield oldfunc(char, *args, **kwargs) return newfunc This is probably a bad example, but I hope it still explains why it should be legal to define generators in context managers - as long as the with block serves its purpose correctly, everything else should still work normally. For those concerned about backwards compatibility: I believe that those who attempt to yield inside a context manager will already discover that results are undefined when doing so; this will simply make it more obvious that suspending execution in a with block is not meant to happen, and convert undefined behavior into a straight-up SyntaxError. What are your thoughts? Sharing, Ken Hilton;

15 20

Syntactic sugar to declare partial functions
by Fabrizio Messina 06 Aug '18

06 Aug '18

Hello, I would like to propose a new method to create a partial function. At the moment we have to load the *partial* function from the *functool* library, and apply it to an existing function, e.g. from functools import partial def add(x: int, y: int) -> int: return x + y add_2 = partial(add, 2) While partial expose the mechanism excellently its instantiation method is, at times, not very friendly, I would like to propose a syntactic sugar to create partial functions, in the case you create a partial function using *curly braces*: def add(x: int, y: int) -> int: return x + y add_2 = add{2} At the moment this causes SyntaxError so the change is retro-compatible. In the case of key word arguments we could have: sort_by_x = sort{key=lambda element: element.x} That could be good as it would be an easy way to pre-load functions without having to eagerly compute it, but without needing to pass the entire function parameters to to other scopes. # prepare the function get_sorted_users: Callable[[], Iterator[User]] = sort{users, key=lambda user : user.creation_date} # continue with job at hand ... # some where else, maybe another process sorted_users = list(get_sorted_users()) Even create a factory method on the fly: @dataclass class Product: name: str category: Category price: Decimal smartphone_factory = Product{category=smartphone_category} Now all this can already be done with partial, but adding this syntactic sugar would reduce the perception of `partial` as an advanced feature, alleviating the use of closures created only for the sake of avoiding an explicit partial. In my opinion this syntactic sugar has a lot of potential adoption seen the general interest in functional programming.

6 7

With expressions
by Ken Hilton 04 Aug '18

04 Aug '18

Hi, I don't know if someone has already suggested this before, but here goes: With expressions allow using the enter/exit semantics of the with statement inside an expression context. Examples: contents = f.read() with open('file') as f #the most obvious one multiplecontents = [f.read() with open(name) as f for name in names] #reading multiple files I don't know if it's worth making the "as NAME" part of the with mandatory in an expression - is this a valid use case? data = database.selectrows() with threadlock Where this would benefit: I think the major use case is `f.read() with open('file') as f`. Previous documentation has suggested `open('file').read()` and rely on garbage collection; as the disadvantages of that became obvious, it transitioned to a method that couldn't be done in an expression: with open('file') as f: contents = f.read() Therefore `f.read() with open('file') as f`, I think, would be much welcomed as the best way to read a file in an expression. For those wondering about the scope semantics of the "as NAME", I think they would be identical to the scope semantics of the "for" expression - i.e. these are legal: contents = f.read() with open('file') as f grid = [[i] * 4 for i in range(4)] But these are not: contents = f.read() with open('file') as f f.seek(0) grid = [[i] * 4 for i in range(4)] grid[i][i] = 4 Is this a good idea? Are there some subtleties I've failed to explain? Please let me know. Sharing, Ken Hilton

13 26

Re: [Python-ideas] With expressions
by Jeroen Demeyer 04 Aug '18

04 Aug '18

On 2018-08-04 09:43, Steven D'Aprano wrote: > If you want to > propose a general exception suppressing mechanism (aside from the > existing try...except statement) then propose it as an independent PEP. This was already tried in PEP 463 (which IMHO is a pity that it wasn't accepted because it seems really useful).

1 0

Re: [Python-ideas] PEP 505: None-aware operators: operators ?= and ?? and OR
by Jonathan Fine 04 Aug '18

04 Aug '18

Hi > There is a formatted version of this PEP at > https://www.python.org/dev/peps/pep-0505/ I've taken a look at this, and have some comments on the first two examples drawn from standard library code. (And a very grateful +10 for writing a script to find such examples.) I've started a subthread, just to discuss the ?= and ?? operators. And something newish, that I call OR. FIRST EXAMPLE The first example is --- >From bisect.py: def insort_right(a, x, lo=0, hi=None): # ... if hi is None: hi = len(a) --- Here, None is a sentinel value. The simpler code --- hi = hi or len(a) --- fails when hi is zero (or any other value that is False in the boolean context). This can be fixed by introducing a new operator OR which is similar to 'or' but has the semantics this example requires. Thus, given OR we can write --- hi = hi OR len(a) --- where (A OR B) returns A if A is not None, otherwise it returns B. (Recall that (A or B) returns A if bool(A), otherwise it returns B.) SECOND EXAMPLE The second example is --- >From calendar.py: encoding = options.encoding if encoding is None: encoding = sys.getdefaultencoding() optdict = dict(encoding=encoding, css=options.css) --- Once we have OR we would write this as --- encoding = encoding OR sys.getdefaultencoding() optdict = dict(encoding=encoding, css=options.css) --- And from here we can condense into a single (longish) line: --- optdict = dict(encoding=encoding OR sys.getdefaultencoding(), css=options.css) -- SUMMARY Here, for reference, are the suggestions using ?= and ?? in PEP 505. --- hi ??= len(a) --- optdict = dict(encoding=encoding ?? sys.getdefaultencoding(), css=options.css) --- Comments ?? suggestions. For example, would a None-aware AND operator be useful? -- Jonathan

17 28

Change repr of collections.OrderedDict to be more dict-like
by Miro Hrončok 02 Aug '18

02 Aug '18

Hi, now when dicts are sorted, can we change the repr of collections.OrderedDict to look like dict? I mean from: OrderedDict([('a', '1'), ('b', '2')]) To: OrderedDict({'a': '1', 'b': '2'}) I assume the current list-of-tuples repr is there so copy pasting the repr to the interpreter results in the same OrderedDict. When dicts preserve insertion order, this now works with dict. I consider the dict-based repr much easier to read. Thoughts? -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

13 23

PEP 505: None-aware operators
by Steve Dower 01 Aug '18

01 Aug '18

Possibly this is exactly the wrong time to propose the next big syntax change, since we currently have nobody to declare on it, but since we're likely to argue for a while anyway it probably can't hurt (and maybe this will become the test PEP for whoever takes the reins?). FWIW, Guido had previously indicated that he was generally favourable towards most of this proposal, provided we could figure out coherent semantics. Last time we tried, that didn't happen, so this time I've made the semantics much more precise, have implemented and verified them, and made much stronger statements about why we are proposing these. Additional thanks to Mark Haase for writing most of the PEP. All the fair and balanced parts are his - all the overly strong opinions are mine. Also thanks to Nick Coghlan for writing PEPs 531 and 532 last time we went through this - if you're unhappy with "None" being treated as a special kind of value, I recommend reading those before you start repeating them. There is a formatted version of this PEP at https://www.python.org/dev/peps/pep-0505/ My current implementation is at https://github.com/zooba/cpython/tree/pep-505 (though I'm considering removing some of the new opcodes I added and just generating more complex code - in any case, let's get hung up on the proposal rather than the implementation :) ) Let the discussions begin! --- PEP: 505 Title: None-aware operators Version: $Revision$ Last-Modified: $Date$ Author: Mark E. Haase <mehaase(a)gmail.com>, Steve Dower <steve.dower(a)python.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 18-Sep-2015 Python-Version: 3.8 Abstract ======== Several modern programming languages have so-called "``null``-coalescing" or "``null``- aware" operators, including C# [1]_, Dart [2]_, Perl, Swift, and PHP (starting in version 7). These operators provide syntactic sugar for common patterns involving null references. * The "``null``-coalescing" operator is a binary operator that returns its left operand if it is not ``null``. Otherwise it returns its right operand. * The "``null``-aware member access" operator accesses an instance member only if that instance is non-``null``. Otherwise it returns ``null``. (This is also called a "safe navigation" operator.) * The "``null``-aware index access" operator accesses an element of a collection only if that collection is non-``null``. Otherwise it returns ``null``. (This is another type of "safe navigation" operator.) This PEP proposes three ``None``-aware operators for Python, based on the definitions and other language's implementations of those above. Specifically: * The "``None`` coalescing`` binary operator ``??`` returns the left hand side if it evaluates to a value that is not ``None``, or else it evaluates and returns the right hand side. A coalescing ``??=`` augmented assignment operator is included. * The "``None``-aware attribute access" operator ``?.`` evaluates the complete expression if the left hand side evaluates to a value that is not ``None`` * The "``None``-aware indexing" operator ``?[]`` evaluates the complete expression if the left hand site evaluates to a value that is not ``None`` Syntax and Semantics ==================== Specialness of ``None`` ----------------------- The ``None`` object denotes the lack of a value. For the purposes of these operators, the lack of a value indicates that the remainder of the expression also lacks a value and should not be evaluated. A rejected proposal was to treat any value that evaluates to false in a Boolean context as not having a value. However, the purpose of these operators is to propagate the "lack of value" state, rather that the "false" state. Some argue that this makes ``None`` special. We contend that ``None`` is already special, and that using it as both the test and the result of these operators does not change the existing semantics in any way. See the `Rejected Ideas`_ section for discussion on the rejected approaches. Grammar changes --------------- The following rules of the Python grammar are updated to read:: augassign: ('+=' | '-=' | '*=' | '@=' | '/=' | '%=' | '&=' | '|=' | '^=' | '<<=' | '>>=' | '**=' | '//=' | '??=') power: coalesce ['**' factor] coalesce: atom_expr ['??' factor] atom_expr: ['await'] atom trailer* trailer: ('(' [arglist] ')' | '[' subscriptlist ']' | '?[' subscriptlist ']' | '.' NAME | '?.' NAME) Inserting the ``coalesce`` rule in this location ensures that expressions resulting in ``None`` are natuarlly coalesced before they are used in operations that would typically raise ``TypeError``. Like ``and`` and ``or`` the right-hand expression is not evaluated until the left-hand side is determined to be ``None``. For example:: a, b = None, None def c(): return None def ex(): raise Exception() (a ?? 2 ** b ?? 3) == a ?? (2 ** (b ?? 3)) (a * b ?? c // d) == a * (b ?? c) // d (a ?? True and b ?? False) == (a ?? True) and (b ?? False) (c() ?? c() ?? True) == True (True ?? ex()) == True (c ?? ex)() == c() Augmented coalescing assignment only rebinds the name if its current value is ``None``. If the target name already has a value, the right-hand side is not evaluated. For example:: a = None b = '' c = 0 a ??= 'value' b ??= undefined_name c ??= shutil.rmtree('/') # don't try this at home, kids assert a == 'value' assert b == '' assert c == '0' and any(os.scandir('/')) Adding new trailers for the other ``None``-aware operators ensures that they may be used in all valid locations for the existing equivalent operators, including as part of an assignment target (more details below). As the existing evaluation rules are not directly embedded in the grammar, we specify the required changes here. Assume that the ``atom`` is always successfully evaluated. Each ``trailer`` is then evaluated from left to right, applying its own parameter (either its arguments, subscripts or attribute name) to produce the value for the next ``trailer``. Finally, if present, ``await`` is applied. For example, ``await a.b(c).d[e]`` is currently parsed as ``['await', 'a', '.b', '(c)', '.d', '[e]']`` and evaluated:: _v = a _v = _v.b _v = _v(c) _v = _v.d _v = _v[e] await _v When a ``None``-aware operator is present, the left-to-right evaluation may be short-circuited. For example, ``await a?.b(c).d?[e]`` is evaluated:: _v = a if _v is not None: _v = _v.b _v = _v(c) _v = _v.d if _v is not None: _v = _v[e] await _v .. note:: ``await`` will almost certainly fail in this context, as it would in the case where code attempts ``await None``. We are not proposing to add a ``None``-aware ``await`` keyword here, and merely include it in this example for completeness of the specification, since the ``atom_expr`` grammar rule includes the keyword. If it were in its own rule, we would have never mentioned it. Parenthesised expressions are handled by the ``atom`` rule (not shown above), which will implicitly terminate the short-circuiting behaviour of the above transformation. For example, ``(a?.b ?? c).d?.e`` is evaluated as:: # a?.b _v = a if _v is not None: _v = _v.b # ... ?? c if _v is None: _v = c # (...).d?.e _v = _v.d if _v is not None: _v = _v.e When used as an assignment target, the ``None``-aware operations may only be used in a "load" context. That is, ``a?.b = 1`` and ``a?[b] = 1`` will raise ``SyntaxError``. Use earlier in the expression (``a?.b.c = 1``) is permitted, though unlikely to be useful unless combined with a coalescing operation:: (a?.b ?? d).c = 1 Examples ======== This section presents some examples of common ``None`` patterns and shows what conversion to use ``None``-aware operators may look like. Standard Library ---------------- Using the ``find-pep505.py`` script[3]_ an analysis of the Python 3.7 standard library discovered up to 678 code snippets that could be replaced with use of one of the ``None``-aware operators:: $ find /usr/lib/python3.7 -name '*.py' | xargs python3.7 find-pep505.py <snip> Total None-coalescing `if` blocks: 449 Total [possible] None-coalescing `or`: 120 Total None-coalescing ternaries: 27 Total Safe navigation `and`: 13 Total Safe navigation `if` blocks: 61 Total Safe navigation ternaries: 8 Some of these are shown below as examples before and after converting to use the new operators. From ``bisect.py``:: def insort_right(a, x, lo=0, hi=None): # ... if hi is None: hi = len(a) # ... After updating to use the ``??=`` augmented assignment statement:: def insort_right(a, x, lo=0, hi=None): # ... hi ??= len(a) # ... From ``calendar.py``:: encoding = options.encoding if encoding is None: encoding = sys.getdefaultencoding() optdict = dict(encoding=encoding, css=options.css) After updating to use the ``??`` operator:: optdict = dict(encoding=encoding ?? sys.getdefaultencoding(), css=options.css) From ``dis.py``:: def _get_const_info(const_index, const_list): argval = const_index if const_list is not None: argval = const_list[const_index] return argval, repr(argval) After updating to use the ``?[]`` and ``??`` operators:: def _get_const_info(const_index, const_list): argval = const_list?[const_index] ?? const_index return argval, repr(argval) From ``inspect.py``:: for base in object.__bases__: for name in getattr(base, "__abstractmethods__", ()): value = getattr(object, name, None) if getattr(value, "__isabstractmethod__", False): return True After updating to use the ``?.`` operator (and deliberately not converting to use ``any()``):: for base in object.__bases__: for name in base?.__abstractmethods__ ?? (): if object?.name?.__isabstractmethod__: return True From ``os.py``:: if entry.is_dir(): dirs.append(name) if entries is not None: entries.append(entry) else: nondirs.append(name) After updating to use the ``?.`` operator:: if entry.is_dir(): dirs.append(name) entries?.append(entry) else: nondirs.append(name) jsonify ------- This example is from a Python web crawler that uses the Flask framework as its front-end. This function retrieves information about a web site from a SQL database and formats it as JSON to send to an HTTP client:: class SiteView(FlaskView): @route('/site/<id_>', methods=['GET']) def get_site(self, id_): site = db.query('site_table').find(id_) return jsonify( first_seen=site.first_seen.isoformat() if site.first_seen is not None else None, id=site.id, is_active=site.is_active, last_seen=site.last_seen.isoformat() if site.last_seen is not None else None, url=site.url.rstrip('/') ) Both ``first_seen`` and ``last_seen`` are allowed to be ``null`` in the database, and they are also allowed to be ``null`` in the JSON response. JSON does not have a native way to represent a ``datetime``, so the server's contract states that any non-``null`` date is represented as an ISO-8601 string. Without knowing the exact semantics of the ``first_seen`` and ``last_seen`` attributes, it is impossible to know whether the attribute can be safely or performantly accessed multiple times. One way to fix this code is to replace each conditional expression with an explicit value assignment and a full ``if``/``else`` block:: class SiteView(FlaskView): @route('/site/<id_>', methods=['GET']) def get_site(self, id_): site = db.query('site_table').find(id_) first_seen_dt = site.first_seen if first_seen_dt is None: first_seen = None else: first_seen = first_seen_dt.isoformat() last_seen_dt = site.last_seen if last_seen_dt is None: last_seen = None else: last_seen = last_seen_dt.isoformat() return jsonify( first_seen=first_seen, id=site.id, is_active=site.is_active, last_seen=last_seen, url=site.url.rstrip('/') ) This adds ten lines of code and four new code paths to the function, dramatically increasing the apparent complexity. Rewriting using the ``None``-aware attribute operator results in shorter code with more clear intent:: class SiteView(FlaskView): @route('/site/<id_>', methods=['GET']) def get_site(self, id_): site = db.query('site_table').find(id_) return jsonify( first_seen=site.first_seen?.isoformat(), id=site.id, is_active=site.is_active, last_seen=site.last_seen?.isoformat(), url=site.url.rstrip('/') ) Grab ---- The next example is from a Python scraping library called `Grab <https://github.com/lorien/grab/blob/4c95b18dcb0fa88eeca81f5643c0ebfb114bf72… ab/upload.py>`_:: class BaseUploadObject(object): def find_content_type(self, filename): ctype, encoding = mimetypes.guess_type(filename) if ctype is None: return 'application/octet-stream' else: return ctype class UploadContent(BaseUploadObject): def __init__(self, content, filename=None, content_type=None): self.content = content if filename is None: self.filename = self.get_random_filename() else: self.filename = filename if content_type is None: self.content_type = self.find_content_type(self.filename) else: self.content_type = content_type class UploadFile(BaseUploadObject): def __init__(self, path, filename=None, content_type=None): self.path = path if filename is None: self.filename = os.path.split(path)[1] else: self.filename = filename if content_type is None: self.content_type = self.find_content_type(self.filename) else: self.content_type = content_type This example contains several good examples of needing to provide default values. Rewriting to use conditional expressions reduces the overall lines of code, but does not necessarily improve readability:: class BaseUploadObject(object): def find_content_type(self, filename): ctype, encoding = mimetypes.guess_type(filename) return 'application/octet-stream' if ctype is None else ctype class UploadContent(BaseUploadObject): def __init__(self, content, filename=None, content_type=None): self.content = content self.filename = (self.get_random_filename() if filename is None else filename) self.content_type = (self.find_content_type(self.filename) if content_type is None else content_type) class UploadFile(BaseUploadObject): def __init__(self, path, filename=None, content_type=None): self.path = path self.filename = (os.path.split(path)[1] if filename is None else filename) self.content_type = (self.find_content_type(self.filename) if content_type is None else content_type) The first ternary expression is tidy, but it reverses the intuitive order of the operands: it should return ``ctype`` if it has a value and use the string literal as fallback. The other ternary expressions are unintuitive and so long that they must be wrapped. The overall readability is worsened, not improved. Rewriting using the ``None`` coalescing operator:: class BaseUploadObject(object): def find_content_type(self, filename): ctype, encoding = mimetypes.guess_type(filename) return ctype ?? 'application/octet-stream' class UploadContent(BaseUploadObject): def __init__(self, content, filename=None, content_type=None): self.content = content self.filename = filename ?? self.get_random_filename() self.content_type = content_type ?? self.find_content_type(self.filename) class UploadFile(BaseUploadObject): def __init__(self, path, filename=None, content_type=None): self.path = path self.filename = filename ?? os.path.split(path)[1] self.content_type = content_type ?? self.find_content_type(self.filename) This syntax has an intuitive ordering of the operands. In ``find_content_type``, for example, the preferred value ``ctype`` appears before the fallback value. The terseness of the syntax also makes for fewer lines of code and less code to visually parse, and reading from left-to-right and top-to-bottom more accurately follows the execution flow. Rejected Ideas ============== The first three ideas in this section are oft-proposed alternatives to treating ``None`` as special. For further background on why these are rejected, see their treatment in `PEP 531 <https://www.python.org/dev/peps/pep-0531/>`_ and `PEP 532 <https://www.python.org/dev/peps/pep-0532/>`_ and the associated discussions. No-Value Protocol ----------------- The operators could be generalised to user-defined types by defining a protocol to indicate when a value represents "no value". Such a protocol may be a dunder method ``__has_value__(self)` that returns ``True`` if the value should be treated as having a value, and ``False`` if the value should be treated as no value. With this generalization, ``object`` would implement a dunder method equivalent to this:: def __has_value__(self): return True ``NoneType`` would implement a dunder method equivalent to this:: def __has_value__(self): return False In the specification section, all uses of ``x is None`` would be replaced with ``not x.__has_value__()``. This generalization would allow for domain-specific "no-value" objects to be coalesced just like ``None``. For example the ``pyasn1`` package has a type called ``Null`` that represents an ASN.1 ``null``:: >>> from pyasn1.type import univ >>> univ.Null() ?? univ.Integer(123) Integer(123) Similarly, values such as ``math.nan`` and ``NotImplemented`` could be treated as representing no value. However, the "no-value" nature of these values is domain-specific, which means they *should* be treated as a value by the language. For example, ``math.nan.imag`` is well defined (it's ``0.0``), and so short-circuiting ``math.nan?.imag`` to return ``math.nan`` would be incorrect. As ``None`` is already defined by the language as being the value that represents "no value", and the current specification would not preclude switching to a protocol in the future (though changes to built-in objects would not be compatible), this idea is rejected for now. Boolean-aware operators ----------------------- This suggestion is fundamentally the same as adding a no-value protocol, and so the discussion above also applies. Similar behavior to the ``??`` operator can be achieved with an ``or`` expression, however ``or`` checks whether its left operand is false-y and not specifically ``None``. This approach is attractive, as it requires fewer changes to the language, but ultimately does not solve the underlying problem correctly. Assuming the check is for truthiness rather than ``None``, there is no longer a need for the ``??`` operator. However, applying this check to the ``?.`` and ``?[]`` operators prevents perfectly valid operations applying Consider the following example, where ``get_log_list()`` may return either a list containing current log messages (potentially empty), or ``None`` if logging is not enabled:: lst = get_log_list() lst?.append('A log message') If ``?.`` is checking for true values rather than specifically ``None`` and the log has not been initialized with any items, no item will ever be appended. This violates the obvious intent of the code, which is to append an item. The ``append`` method is available on an empty list, as are all other list methods, and there is no reason to assume that these members should not be used because the list is presently empty. Further, there is no sensible result to use in place of the expression. A normal ``lst.append`` returns ``None``, but under this idea ``lst?.append`` may result in either ``[]`` or ``None``, depending on the value of ``lst``. As with the examples in the previous section, this makes no sense. As checking for truthiness rather than ``None`` results in apparently valid expressions no longer executing as intended, this idea is rejected. Exception-aware operators ------------------------- Arguably, the reason to short-circuit an expression when ``None`` is encountered is to avoid the ``AttributeError`` or ``TypeError`` that would be raised under normal circumstances. As an alternative to testing for ``None``, the ``?.`` and ``?[]`` operators could instead handle ``AttributeError`` and ``TypeError`` raised by the operation and skip the remainder of the expression. This produces a transformation for ``a?.b.c?.d.e`` similar to this:: _v = a try: _v = _v.b except AttributeError: pass else: _v = _v.c try: _v = _v.d except AttributeError: pass else: _v = _v.e One open question is which value should be returned as the expression when an exception is handled. The above example simply leaves the partial result, but this is not helpful for replacing with a default value. An alternative would be to force the result to ``None``, which then raises the question as to why ``None`` is special enough to be the result but not special enough to be the test. Secondly, this approach masks errors within code executed implicitly as part of the expression. For ``?.``, any ``AttributeError`` within a property or ``__getattr__`` implementation would be hidden, and similarly for ``?[]`` and ``__getitem__`` implementations. Similarly, simple typing errors such as ``{}?.ietms()`` could go unnoticed. Existing conventions for handling these kinds of errors in the form of the ``getattr`` builtin and the ``.get(key, default)`` method pattern established by ``dict`` show that it is already possible to explicitly use this behaviour. As this approach would hide errors in code, it is rejected. ``None``-aware Function Call ---------------------------- The ``None``-aware syntax applies to attribute and index access, so it seems natural to ask if it should also apply to function invocation syntax. It might be written as ``foo?()``, where ``foo`` is only called if it is not None. This has been deferred on the basis of the proposed operators being intended to aid traversal of partially populated hierarchical data structures, *not* for traversal of arbitrary class hierarchies. This is reflected in the fact that none of the other mainstream languages that already offer this syntax have found it worthwhile to support a similar syntax for optional function invocations. A workaround similar to that used by C# would be to write ``maybe_none?.__call__(arguments)``. If the callable is ``None``, the expression will not be evaluated. (The C# equivalent uses ``?.Invoke()`` on its callable type.) ``?`` Unary Postfix Operator ---------------------------- To generalize the ``None``-aware behavior and limit the number of new operators introduced, a unary, postfix operator spelled ``?`` was suggested. The idea is that ``?`` might return a special object that could would override dunder methods that return ``self``. For example, ``foo?`` would evaluate to ``foo`` if it is not ``None``, otherwise it would evaluate to an instance of ``NoneQuestion``:: class NoneQuestion(): def __call__(self, *args, **kwargs): return self def __getattr__(self, name): return self def __getitem__(self, key): return self With this new operator and new type, an expression like ``foo?.bar[baz]`` evaluates to ``NoneQuestion`` if ``foo`` is None. This is a nifty generalization, but it's difficult to use in practice since most existing code won't know what ``NoneQuestion`` is. Going back to one of the motivating examples above, consider the following:: >>> import json >>> created = None >>> json.dumps({'created': created?.isoformat()})`` The JSON serializer does not know how to serialize ``NoneQuestion``, nor will any other API. This proposal actually requires *lots of specialized logic* throughout the standard library and any third party library. At the same time, the ``?`` operator may also be **too general**, in the sense that it can be combined with any other operator. What should the following expressions mean?:: >>> x? + 1 >>> x? -= 1 >>> x? == 1 >>> ~x? This degree of generalization is not useful. The operators actually proposed herein are intentionally limited to a few operators that are expected to make it easier to write common code patterns. Built-in ``maybe`` ------------------ Haskell has a concept called `Maybe <https://wiki.haskell.org/Maybe>`_ that encapsulates the idea of an optional value without relying on any special keyword (e.g. ``null``) or any special instance (e.g. ``None``). In Haskell, the purpose of ``Maybe`` is to avoid separate handling of "something" and nothing". A Python package called `pymaybe <https://pypi.org/p/pymaybe/>`_ provides a rough approximation. The documentation shows the following example:: >>> maybe('VALUE').lower() 'value' >>> maybe(None).invalid().method().or_else('unknown') 'unknown' The function ``maybe()`` returns either a ``Something`` instance or a ``Nothing`` instance. Similar to the unary postfix operator described in the previous section, ``Nothing`` overrides dunder methods in order to allow chaining on a missing value. Note that ``or_else()`` is eventually required to retrieve the underlying value from ``pymaybe``'s wrappers. Furthermore, ``pymaybe`` does not short circuit any evaluation. Although ``pymaybe`` has some strengths and may be useful in its own right, it also demonstrates why a pure Python implementation of coalescing is not nearly as powerful as support built into the language. The idea of adding a builtin ``maybe`` type to enable this scenario is rejected. Just use a conditional expression --------------------------------- Another common way to initialize default values is to use the ternary operator. Here is an excerpt from the popular `Requests package <https://github.com/kennethreitz/requests/blob/14a555ac716866678bf17e43e2323… a8149f5/requests/models.py#L212>`_:: data = [] if data is None else data files = [] if files is None else files headers = {} if headers is None else headers params = {} if params is None else params hooks = {} if hooks is None else hooks This particular formulation has the undesirable effect of putting the operands in an unintuitive order: the brain thinks, "use ``data`` if possible and use ``[]`` as a fallback," but the code puts the fallback *before* the preferred value. The author of this package could have written it like this instead:: data = data if data is not None else [] files = files if files is not None else [] headers = headers if headers is not None else {} params = params if params is not None else {} hooks = hooks if hooks is not None else {} This ordering of the operands is more intuitive, but it requires 4 extra characters (for "not "). It also highlights the repetition of identifiers: ``data if data``, ``files if files``, etc. When written using the ``None`` coalescing operator, the sample reads:: data = data ?? [] files = files ?? [] headers = headers ?? {} params = params ?? {} hooks = hooks ?? {} References ========== .. [1] C# Reference: Operators (https://msdn.microsoft.com/en-us/library/6a71f45d.aspx) .. [2] A Tour of the Dart Language: Operators (https://www.dartlang.org/docs/dart-up-and-running/ch02.html#operators) .. [3] Associated scripts (https://github.com/python/peps/tree/master/pep-0505/) Copyright ========= This document has been placed in the public domain.

44 273