Mailman 3 May 2020 - Python-ideas

Resetting peak memory metric in tracemalloc without touching other traces
by wilson.huon＠gmail.com May 15, 2020

May 15, 2020

Hi, It would be helpful for us if tracemalloc had a function that reset the peak memory usage counter, without clearing the current traces. At the moment, I don't think there's a way to find the peak memory of a subset of the code since the initial tracemalloc.start() call, without calling tracemalloc.clear_traces(). The latter disturbs other parts of the tracing. Specifically, it might be a function like (pseudo-implementation): def reset_peak_memory(): # in _tracemalloc.c tracemalloc_peak_trace_memory = tracemalloc_traced_memory; This would allow easily determining the peak memory usage of a specific piece of code, without disturbing all of the traces. For example, the following would set specific_peak to the highest size of traced memory of just line X: tracemalloc.start() # ... code where allocations matter, but the peak does not ... peak_memory_doesnt_matter() tracemalloc.reset_peak_memory() peak_memory_is_important() # X _, specific_peak = tracemalloc.get_traced_memory() # ... more code with relevant allocations ... peak_memory_doesnt_matter() tracemalloc.stop() As sketched above, the implementation of this should be quite small, with the core being the line mentioned above, plus all the required extras (locking, wrapping, documentation, tests, ...). Thoughts? Full motivation for why we want to do this: In <https://github.com/stellargraph/stellargraph>, we're using the tracemalloc module to understand the memory usage of our core StellarGraph graph class (a nodes-and-edges graph, not a plot, to be clear). It stores some NumPy arrays of feature vectors associated with each node in the graph, along with all of the edge information. Any of these pieces can be large, and we want to keep the resource usage as small as possible. We're monitoring this by instrumenting the construction: start from a raw set of nodes (including potentially large amounts of features) and edges, and build a StellarGraph object, recording some metrics: 1. the time 2. the total memory usage of the graph instance 3. the additional memory usage, that's not shared with the raw data (in particular, if the raw data is 1GB, it's useful to know whether a 1.5GB graph instance consists of 0.5GB of new memory, or 1.5GB of new memory) 4. the peak memory usage during construction 2, 3 and 4 we record using a combination of tracemalloc.take_snapshot() and tracemalloc.get_traced_memory(), something like: def diff(after, before): return sum(elem.size_diff for after.compare_to(before, "lineno")) snap_start = take_snapshot() raw = load_data_from_disk() snap_raw = take_snapshot() # X graph = create_graph(raw) snap_raw_graph = take_snapshot() _, mem_peak = get_traced_memory() # 4 del raw snap_graph = take_snapshot() mem_raw = diff(snap_raw, snap_start) # baseline mem_graph = diff(snap_graph, snap_start) # 2 mem_graph_not_shared = diff(snap_raw_graph, snap_raw) # 3 ('measure_memory' in <https://nbviewer.jupyter.org/github/stellargraph/stellargraph/blob/93fce461…> has all the gory details.) Unfortunately, we want to ignore any peak during data loading: the peak during create_graph is all we care about, even if the overall peak (in data loading) is higher. That is, we want to only consider the peak memory usage after line X. One way to do this would be to call clear_traces() at X, but this invalidates the traces used for the 2 and 3 comparisons. I believe tracemalloc.reset_peak_memory() is the necessary function to call at X. (Why do we want to ignore the peak during data loading? The loading is under the control of a user (of stellargraph) as it's typically done via Pandas or NumPy and those libraries are out of our control and offer a variety of options for tweaking data-loading behavior, whereas the internals of the `StellarGraph` instance are in our control and not as configurable by users.) Thanks, Huon Wilson

3 2

Making asserts non-recoverable.
by chris.the.developer.0＠gmail.com May 14, 2020

May 14, 2020

How adverse would you guys feel about a change to the way asserts are handled so that they are not recoverable? Asserts could latch on to the first failed assert and always throw an AssertionError on subsequent assert statements. Another way would be for subsequent asserts to turn off after the first failure. As long as developers keep the contract that asserts should never fail, they wont notice this change; and it blocks developers from writing code that is dependent on asserts.

10 13

Sanitize filename (path part) 2nd try
by Steve Jorgensen May 13, 2020

May 13, 2020

Based on responses to my previous proposal, I am convinced that it was over-ambitious and not appropriate for inclusion in the Python standard library, so starting over with a more narrowly scoped suggestion. Proposal: Add a new function (possibly `os.path.sanitizepart`) to sanitize a value for use as a single component of a path. In the default case, the value must also not be a reference to the current or parent directory ("." or "..") and must not contain control characters. When an invalid character is encountered, then `ValueError` will be raised in the default case, or the character may be replaced or escaped. When an invalid name is encountered, then `ValueError` will be raised in the default case, or the first character may be replaced, escaped, or prefixed. Control characters (those in the Unicode general category of "C") are treated as invalid by default. After applying any transformations, if the result would still be invalid, then an exception is raised. Proposed function signature: `sanitizepart(name, replace=None, escape=None, prefix=None, flags=0)` When `replace` is supplied, it is used as a replacement for any invalid characters or for the first character of an invalid name. When `prefix` is not also supplied, this is also used as the replacement for the first character of the name if it is invalid, not simply due to containing invalid characters. When `escape` is supplied (typically "%") it is used as the escape character in the same way that "%" is used in URL encoding. When a non-ASCII character is escaped, it is represented as a sequence of encoded bytes/octets. When `prefix` is not also supplied, this is also used to escape the first character of the name if it is invalid, not simply due to containing invalid characters. `replace` and `escape` are mutually exclusive. When `prefix` is supplied (typically "_"), it is prepended the name if it is invalid, not simply due to containing invalid characters. Flags: - path.PERMIT_RELATIVE (1): Permit relative path values ("." "..") - path.PERMIT_CTRL (2): Permit characters in the Unicode general category of "C".

9 17

Sanitize filename (path part)
by Steve Jorgensen May 12, 2020

May 12, 2020

I believe the Python standard library should include a means of sanitizing a filesystem entry, and this should not be something requiring a 3rd party package. One of reasons I think this should be in the standard lib is because that provides a common, simple means for code reviewers and static analysis services such as Veracode to recognize that a value is sanitized in an accepted manner. What I am envisioning is a function (presumably in `os.path` with a signature roughly like {{{ sanitizepart(name, permissive=False, mode=ESCAPE, system=None) }}} When `permissive` is `False`, characters that are generally unsafe are rejected. When `permissive` is `True`, only path separator characters are rejected. Generally unsafe characters besides path separators would include things like a leading ".", any non-printing character, any wildcard, piping and redirection characters, etc. The `mode` argument indicates what to do with unacceptable characters. Escape them (`ESCAPE`), omit them (`OMIT`) or raise an exception (`RAISE`). This could also double as an escape character argument when a string is given. The default escape character should probably be "%" (same as URL encoding). The `system` argument accepts a combination of bit flags indicating what operating system's rules to apply, or `None` meaning to use rules for the current platform. Systems would probably include `SYS_POSIX`, `SYS_WIN`, and `SYS_MISC` where miscellaneous means to enforce rules for all commonly used systems. One example of a distinction is that on a POSIX system, backslash characters are not path separators, but on Windows, both forward and backward slashes are path separators. {{{ from os import path from os.path import sanitizepart print(repr( os.path.sanitizepart('/ABC\\QRS%', system=path.SYS_WIN)) # => '%2fABC%5cQRS%%' os.path.sanitizepart('/ABC\\QRS%', True, mode=path.STRIP, system=path.SYS_POSIX)) # => 'ABC\\QRS%' os.path.sanitizepart('../AB&CD*\x01\n', system=path.SYS_POSIX)) # => '%2e.%2fABC%26CD%2a%01%10' os.path.sanitizepart('../AB&CD*\x01\n', True, system=path.SYS_POSIX)) # => '..%2eAB&CD*\x01\n' }}}

7 19

Equality between some of the indexed collections
by Ahmed Amr May 9, 2020

May 9, 2020

I'd like to take your opinion on modifying some of the indexed collections like tuples, lists, arrays to evaluate its equality to True when having the same items in the same indexes. Currently, when comparing a list of items to an array of the same items for equality (==) it returns False, I'm thinking that it would make sense to return True in that context, as we're comparing item values and we have the same way of indexing both collections, so we can compare item values. So what do you think about applying such behavior on collections that can be indexed the same way such as tuples, lists, and arrays? Example: (Current) import array tuple_ = (1.1, 2.2, 3.3) list_ = [1.1, 2.2, 3.3] array_ = array.array('f', [1.1, 2.2, 3.3]) # all of the following prints False. print(tuple_ == list_) print(tuple_ == array_) print(array_ == list_) Example: (Proposed): All prints above to show True as they are populated with the same data in the same indexes. A Side Note: An extra point to discuss, based on arrays implementation, array_.to_list() would actually get [1.100000023841858, 2.200000047683716, 3.299999952316284] which is not exactly what we've passed as args and this is normal, but I'm thinking about leaving it to the array implementation to encapsulate that implementation and perform exact equality based on passed arguments.

24 104

Introduce 100 more built-in exceptions
by Ram Rachum May 9, 2020

May 9, 2020

Hi, Here's something I wanted in Python for many years. If this has been discussed in the past, please refer me to that discussion. On one hand, it's something that I can't imagine the python-dev community supporting. On the other hand, it would maintain backward compatibility. I wish there were a 100 more built-in exceptions in Python, that will be very specific about what went wrong. If I do this: >>> x, y = range(3) I know it'll raise a ValueError, because I've memorized that, but it did take me a few years to remember where I should expect ValueError and where I should expect TypeError. It would be nice if the operation above raised UnpackingOverflowError, which will be a subclass of UnpackingError, along with UnpackingUnderflowError. UnpackingError can be a subclass of ValueError, for backward compatibility. Similarly, if I did this: >>> def f(x, y): return x + y >>> f(1) I would get a TypeError. Would be a lot cooler if I got MissingArgumentsError, which would be a subclass of SignatureError, which would be a subclass of TypeError. There are 2 reasons I want this: 1. When I'm writing a try..except clause, I want to catch a specific exception like MissingArgumentsError rather than ValueError or TypeError. They're too ubiquitous. I don't want some other unexpected failure producing the same ValueError and triggering my except clause. 2. When I get an error, especially from some shitty corporate system that truncates the traceback, I want to get as many hints as possible about what went wrong. It's true that today, most Python exceptions have good text in their message, like "TypeError: f() missing 1 required positional argument: 'y'". But that isn't guaranteed everywhere, and specific exception types could help. What do you think? Ram.

16 48

General methods
by Serhiy Storchaka May 9, 2020

May 9, 2020

Methods in Python are defined as functions in the class namespace. When call the method of the object, the function will be called with the object as the first argument. And furthermore, unbound methods can be called with passing self as the first argument. For example, str.upper('abc') returns 'ABC'. So class can be considered as a namespace for functions related to objects of the specified type. For methods defined in Python you can pass arbitrary object as self. But in methods defined in C it should be an instance of the class in which the method was defined. On one hand, it is very convenient -- you cab be sure that self is binary compatible with the specified class. On other hand, it restricts you. I propose to add the METH_GENERAL flag, which is applicable to methods as METH_CLASS and METH_STATIC (and is mutually incompatible with them). If it is set, the check for the type of self will be omitted, and you can pass an arbitrary object as the first argument of the unbound method. I have several use cases for this. 1. Bytes and bytearray methods. Bytes and bytearray has a lot of of sequence-like and string-like methods. They also implement the buffer protocol. There are other objects which implement the buffer protocol: memoryview, BytesIO, array.array, mmap.mmap, ctypes arrays, NumPy arrays, but they lack most of these methods. To use these methods (for example index()) you need to copy the content to a bytes or bytearray, that invalidates the purpose of the buffer protocol which was designed to avoid copying of binary data. With general methods we can make bytes.index() be applicable to any object which supports the buffer protocol. bytes.index() and bytearray.index() will be equivalent, but the difference between bytes.split() and bytearray.split() will be in the result type. 2. Set methods. Some set methods accept arbitrary number of arguments and accept not only sets, but any iterables. Sou you can get a union of two lists for example: >>> set().union([1, 2], [2, 3]) {1, 2, 3} It does work because a union with an empty set is a no-op. This trick does not work with other methods. You have to convert the first iterable to set explicitly. >>> set([1, 2]).symmetric_difference([2, 3]) {1, 3} With general methods we can make unbound set methods accepting arbitrary iterables and convert the first one to set implicitly (or avoid creating a set if it is possible). We could use set.symmetric_difference([1, 2], [2, 3]). Maybe there are other use cases. But I do not suggest making all methods of builtin types general: only if there is a more general protocol (as the buffer protocol or the iterator protocol) and there is a profit of using such protocol without creating an object of the corresponding type explicitly. For example, I think that str methods should not be general, despite the fact that any object can be converted to string. Implicit conversions to str does not have profit and may hide bugs. Other example -- float.as_integer_ratio() should not accept int, despite the fact that most function which accept float implicitly convert int to float. It is a lossy conversion for large integers, and the loss will affect the result. I used the term "general methods" to name this feature, but if there are better proposition I will use it.

5 4

Adding a "once" function to functools
by Tom Forbes May 7, 2020

May 7, 2020

Hello, I would like to suggest adding a simple “once” method to functools. As the name suggests, this would be a decorator that would call the decorated function, cache the result and return it with subsequent calls. My rationale for suggesting this addition is twofold: First: It’s fairly common to use `lru_cache()` to implement this behaviour. We use this inside Django (example <https://github.com/django/django/blob/77aa74cb70dd85497dbade6bc0f394aa41e88…>), internally in other projects at my workplace, inside the stdlib itself <https://github.com/python/cpython/blob/2fa67df605e4b0803e7e3aac0b85d851b4b4…> and in numerous other projects. In the first few pages of a Github code search <https://github.com/search?l=Python&q=%22functools.lru_cache%22&type=Code> it is fairly easy to find examples, any decorated method with no parameters is using `lru_cache()` like `once()`. Using lru_cache like this works but it’s not as efficient as it could be - in every case you’re adding lru_cache overhead despite not requiring it. Second: Implementing this in Python, in my opinion, crosses the line of “annoying and non-trivial enough to not want to repeatedly do it”. While a naive (untested) implementation might be: def once(func): sentinel = object() # in case the wrapped method returns None obj = sentinel @functools.wraps(func) def inner(): nonlocal obj, sentinel if obj is sentinel: obj = func() return obj return inner While to the people who are likely going to be reading this mailing this the code above is understandable and potentially even somewhat simple. However to a lot of people who might not have had experience with writing decorators or understand sentinel objects and their use the above code might be incomprehensible. A much more common, and in my opinion worse, implementation that I’ve seen is something along the lines of this: _value = None def get_value(): nonlocal _value if _value is None: _value = some_function() return _value Which is not ideal for obvious reasons. And these are not even including a potentially key feature: locking the wrapped function so that it is only called once if it is invoked from multiple threads at once. So, I’d like to propose adding a `once()` decorator to functools that: 1. Has a C implementation, keeping the speed on-par with `lru_cache()` 2. Ensures that the wrapped function is only called once when invoked by multiple threads For some related discussion about this idea and lru_cache, please see my thread on <https://discuss.python.org/t/reduce-the-overhead-of-functools-lru-cache-for…>discuss.python.org <http://discuss.python.org/>.

22 48

PEP 597 -- Soft deprecation of omitting encoding
by Inada Naoki May 4, 2020

May 4, 2020

The PEP is here: https://www.python.org/dev/peps/pep-0597/ I created a thread on discuss.python.org <https://discuss.python.org/t/pep-597-raise-a-warning-when-encoding-is-omitt…> but I want more feedback. Please reply on this mail or discuss.python.org. ## Summary This PEP doesn't propose to change the default text encoding yet, but introduce warning for code using default text encoding only in dev mode. This opt-in warning is useful for: * Estimate the impact of changing the default text encoding. * Find bugs they use default text encoding where it should be UTF-8. ## PEP Abstract <https://www.python.org/dev/peps/pep-0597/#id6> This PEP proposes: - TextIOWrapper raises a PendingDeprecationWarning when the encoding option is not specified, and dev mode is enabled. - Add encoding="locale" option to TextIOWrapper. It behaves like encoding=None but don't raise a warning. Motivation <https://www.python.org/dev/peps/pep-0597/#id7> Omitting encoding is common mistake <https://www.python.org/dev/peps/pep-0597/#id8> Developers using macOS or Linux may forget that the default encoding is not always UTF-8. For example, long_description = open("README.md").read() in setup.py is a common mistake. Many Windows users can not install the package if there is at least one non-ASCII character (e.g. emoji) in the README.md file. For example, 489 packages of the 4000 most downloaded packages from PyPI used non-ASCII characters in README. And 82 packages of them can not be installed from source package when locale encoding is ASCII. [1 <https://www.python.org/dev/peps/pep-0597/#id2>] They used default encoding to read README or TOML file. Another example is logging.basicConfig(filename="log.txt"). Some users expect UTF-8 is used by default, but locale encoding is used actually. [2 <https://www.python.org/dev/peps/pep-0597/#id3>] Even Python experts assume that default encoding is UTF-8. It creates bugs that happen only on Windows. See [3 <https://www.python.org/dev/peps/pep-0597/#id4>] and [4 <https://www.python.org/dev/peps/pep-0597/#id5>]. Raising a warning when the encoding option is omitted will help to find such mistakes. Prepare to change the default encoding to UTF-8 <https://www.python.org/dev/peps/pep-0597/#id9> We chose to use locale encoding for the default text encoding in Python 3.0. But UTF-8 has been adopted very widely since then. We might change the default text encoding to UTF-8 in the future. But this change will affect many applications and libraries. Many DeprecationWarning will be raised if we start raising the warning by default. It will be too noisy. While this PEP doesn't cover the change, this PEP will help to reduce the number of DeprecationWarning in the future. Specification <https://www.python.org/dev/peps/pep-0597/#id10> Raising a PendingDeprecationWarning <https://www.python.org/dev/peps/pep-0597/#id11> TextIOWrapper raises the PendingDeprecationWarning when the encoding option is omitted, and dev mode is enabled. encoding="locale" option <https://www.python.org/dev/peps/pep-0597/#id12> When encoding="locale" is specified to the TextIOWrapper, it behaves same to encoding=None. In detail, the encoding is chosen by: 1. os.device_encoding(buffer.fileno()) 2. locale.getpreferredencoding(False) This option can be used to suppress the PendingDeprecationWarning. io.text_encoding <https://www.python.org/dev/peps/pep-0597/#id13> TextIOWrapper is used indirectly in most cases. For example, open, and pathlib.Path.read_text() use it. Warning to these functions doesn't make sense. Callers of these functions should be warned instead. io.text_encoding(encoding, stacklevel=1) is a helper function for it. Pure Python implementation will be like this: def text_encoding(encoding, stacklevel=1): """ Helper function to choose the text encoding. When encoding is not None, just return it. Otherwise, return the default text encoding ("locale" for now), and raise a PendingDeprecationWarning in dev mode. This function can be used in APIs having encoding=None option. But please consider encoding="utf-8" for new APIs. """ if encoding is None: if sys.flags.dev_mode: import warnings warnings.warn( "'encoding' option is not specified. The default encoding " "will be changed to 'utf-8' in the future", PendingDeprecationWarning, stacklevel + 2) encoding = "locale" return encoding pathlib.Path.read_text() can use this function like this: def read_text(self, encoding=None, errors=None): """ Open the file in text mode, read it, and close the file. """ encoding = io.text_encoding(encoding) with self.open(mode='r', encoding=encoding, errors=errors) as f: return f.read() subprocess module doesn't warn <https://www.python.org/dev/peps/pep-0597/#id14> While the subprocess module uses TextIOWrapper, it doesn't raise PendingDeprecationWarning. It uses the "locale" encoding by default. Rationale <https://www.python.org/dev/peps/pep-0597/#id15> "locale" is not a codec alias <https://www.python.org/dev/peps/pep-0597/#id16> We don't add the "locale" to the codec alias because locale can be changed in runtime. Additionally, TextIOWrapper checks os.device_encoding() when encoding=None. This behavior can not be implemented in the codec. Use a PendingDeprecationWarning <https://www.python.org/dev/peps/pep-0597/#id17> This PEP doesn't make decision about changing default text encoding. So we use PendingDeprecationWarning instead of DeprecationWarning for now. Raise warning only in dev mode <https://www.python.org/dev/peps/pep-0597/#id18> This PEP will produce a huge amount of PendingDeprecationWarning. It will be too noisy for most Python developers. We need to fix warnings in standard library, pip, and major dev tools like pytest before raise this warning by default. subprocess module doesn't warn <https://www.python.org/dev/peps/pep-0597/#id19> The default encoding for PIPE is relating to the encoding of the stdio. It should be discussed later. Reference Implementation <https://www.python.org/dev/peps/pep-0597/#id20> https://github.com/python/cpython/pull/19481 References <https://www.python.org/dev/peps/pep-0597/#id21> [1] "Packages can't be installed when encoding is not UTF-8" ( https://github.com/methane/pep597-pypi-ascii) [2] "Logging - Inconsistent behaviour when handling unicode" ( https://bugs.python.org/issue37111) [3] Packaging tutorial in packaging.python.org didn't specify encoding to read a README.md (https://github.com/pypa/packaging.python.org/pull/682) [4] json.tool had used locale encoding to read JSON files. ( https://bugs.python.org/issue33684) Copyright <https://www.python.org/dev/peps/pep-0597/#id22> This document has been placed in the public domain. Source: https://github.com/python/peps/blob/master/pep-0597.rst -- Inada Naoki <songofacandy(a)gmail.com>

1 0

Re: Adding a "once" function to functools
by Tom Forbes May 2, 2020

May 2, 2020

> Thread 2 wakes up with the lock, calls the function, fills the cache, and releases the lock. What exactly would the issue be with this: ``` import functools from threading import Lock def once(func): sentinel = object() cache = sentinel lock = Lock() @functools.wraps(func) def _wrapper(): nonlocal cache, lock, sentinel if cache is sentinel: with lock: if cache is sentinel: cache = func() return cache return _wrapper ``` It ensures it’s invoked once, it’s fast and it doesn’t require acquiring a lock needlessly. Always wrapping a lock around a static value to account for a small edge case seems redundant, more than “it’s usually not optimal”. It’s never optimal: in multi-threaded cases you’re causing needless contention, and in single threaded cases all your calls are now slower. Seems generally more correct, even in single threaded cases, to pay the overhead only in the first call if you want `call_once` semantics. Which is why you would be using `call_once` in the first place? > On 29 Apr 2020, at 18:51, Andrew Barnert <abarnert(a)yahoo.com> wrote: > > On Apr 29, 2020, at 00:34, Tom Forbes <tom(a)tomforb.es> wrote: >> >> It’s not quite that easy, either you needlessly lock all calls or you end up invoking it twice. >> >> What you want is to acquire a lock if the cache is empty, check if another thread has filled the cache while you where waiting on the lock, call the function, fill the cache and return. > > Thread 1 sees the cache is empty, so it acquires the lock, runs the function. > > Thread 2 sees the cache is empty, so it tries to acquire the cache and blocks. > > Thread 1 finishes the function, fills the cache, and releases the lock. > > Thread 2 wakes up with the lock, calls the function, fills the cache, and releases the lock. > > You’ve now run the function twice instead of once, and the first caller got a different value than the one every subsequent caller can get. > > There are actually plenty of cases where this is acceptable (the function is safe, and idempotent, and while it’s slow it’s not so slow that occasionally running it twice is worse than locking a zillion times)—but in those cases, just not using a lock is even better. In fact, if you can compare-and-swap at the end (which requires a lock to simulate in Python, but it’s a much more granular one than locking over the whole expensive function call), you can even use it in more cases (in particular, where the function isn’t idempotent but you need to value to be). > > This is exactly why users should be encouraged to just lock the whole thing. It’s always correct. It’s usually not optimal, but often good enough. The one really common and simple case where it’s not good enough is single-threaded code. Anything else where you need to optimize, you really do need to understand the pattern you’re trying to optimize for (and the environment(s) you’re running under) and write your own function. > >

4 7