Mailman 3 May 2017 - Python-ideas

add a LogReader to the logging module
by terji78＠gmail.com 11 May '17

11 May '17

I get a message back that I'm not subscribed to the mailing list, but see my message in google groups. My sincerest apologies in advance, if this appears several times for you. Anyways: The logging module has an easy-to-setup logger: import logging logger = logging.getLogger(__name__) and you just log away. Easy. However, it's quite a bit more difficult to set up log readers, requiring IMO an unreasonably number of lines of code: import logging logger = logging.getLogger('mypackage') logger.setLevel(logging.DEBUG) handler = logging.StreamHandler() formatter = logging.Formatter('%( levelname)s:%(name)s:%(message)s') handler.setFormatter(formatter) logger.addHandler(handler) I propose adding a function that sets up a log reader with sensible defaults, but allowing customization. So, I propose a ``logging.getLogReader`` in the logging module to mirror ``logging.getLogger``. So, to use this, in your main.py, you'd typically just do: import logging log_reader = logging.getLogReader('mypackage.models') and you'd get all log output from mypackage.models with sensible defaults set. Much easier. You could also set up it up in more detail, e.g.: log_reader = logging.getLogReader('mypackage.models', level='debug', format='%(levelname)s | %(filename)s| line %(lineno)s | %(message)s' ) For a specific proposal, see: https://gist.github.com/topper-123/85e27ffe261850eed150eac53d61b82d Because it's just a logger, log_reader can be further customized as necessary. In summary, I think that today it's unneccesarily complex to set up a log reader and the proposed function serves a general enough need, that it - or something similar - should be in the logging module. Thoughts?

3 3

Add shutil.ignore_patterns() to shutil.rmtree()
by George Fischhof 10 May '17

10 May '17

Hi Folks, I have a task to synchronize folders but some files should be remained untouched. I think this is a very common task. I found that shutil.copytree() has ignore_patterns() but rmtree() has not. So here comes my idea: add ignore_patterns() to rmtree() it is a good feature and makes the functions symmetric. BR, George

3 8

Add a .chunks() method to sequences
by robert.hoelzl＠posteo.de 05 May '17

05 May '17

Steven D’Aprano was giving me an idea (in the bytes.hex delimiter discussion): I had very often the use case that I want to split sequences into subsequences of same size. How about adding a chunks() and rchunks() function to sequences: [1,2,3,4,5,6,7].chunks(3) => [[1,2,3], [4,5,6], [7]] "1234“.chunks(2) => [“12“, “34“] (this could then be used to emulate stevens proposal: “ “.join(“1234567“.chunks(2)) => “12 34 56 7“) robert

6 7

Storing a reference to the function object (if any) inside frame objects
by Nathaniel Smith 04 May '17

04 May '17

Hi all, Currently, given a frame object (e.g. from sys._getframe or inspect.getouterframes), there's no way to get back to the function object that created it. This creates an obstacle for various sorts of introspection. In particular, in the unusual but real situation where you need to "mark" a function in a way that can be detected via stack introspection, then the natural way to do that – attaching an attribute to the function object – doesn't work. Instead, you have to mangle the frame's locals. Example: In pytest, if you want to mark a function as uninteresting and not worth showing in tracebacks (a common situation for assertion helpers), then you can't do this with a decorator like @pytest.tracebackhide; instead, you have to set a __magic__ local variable: https://docs.pytest.org/en/latest/example/simple.html#writing-well-integrat… Example: In trio, I want to mark certain Python functions as "protected" from receiving KeyboardInterrupt exceptions – kind of like how CPython doesn't allow KeyboardInterrupt in internal functions [1]. If it were possible for my signal handler to get at function objects when it walks the stack, then the necessary decorator would be something like: def keyboard_interrupt_protected(fn): fn._trio_keyboard_interrupt_protected = True return fn Instead right now it's this mess: https://github.com/python-trio/trio/blob/64119b12309ffeaf3a35622ef08d3b03e4… And worse, the current mess has a race condition, because a KeyboardInterrupt could arrive while we're in the middle of executing the code to enable the KeyboardInterrupt protection. OTOH if this information was carried on the function object, the protection would be be automatically set up when the frame was created. Another benefit is that it would make it easy to include __qualname__'s in exception tracebacks, which is often nice. For example, here's a current traceback: File "/home/njs/trio/trio/_sync.py", line 374, in acquire_nowait return self._lock.acquire_nowait() File "/home/njs/trio/trio/_sync.py", line 277, in acquire_nowait raise WouldBlock And here's what it could look like if the traceback machinery could access frame.f_func.__qualname__: File "/home/njs/trio/trio/_sync.py", line 374, in Condition.acquire_nowait return self._lock.acquire_nowait() File "/home/njs/trio/trio/_sync.py", line 277, in Lock.acquire_nowait raise WouldBlock Here's some semi-terrifying code attempting to work around this issue: https://stackoverflow.com/questions/14817788/python-traceback-with-module-n… To be fair, this last issue could also be solved by adding a f_qualname field to frame objects, like generators and coroutines already have. But another way to think about it is that if we had the whole function object then generators and coroutines wouldn't have to track the qualname separately anymore :-). Q: But what about frames that don't have any associated function object, like those created by exec()? A: Then frame.f_func would be NULL/None, no biggie. Q: Won't this increase memory usage? A: In principle this could increase memory usage by keeping function objects alive longer than otherwise, but I think the effect should be pretty minimal. It doesn't introduce any reference loops, and the vast majority of function objects outlive their frames in any case. Specifically I think the only (?) case where it would make a difference is if you create a new function and then call it immediately without storing it in a temporary, like: (lambda: 1)(). So basically: it's a bit obscure, but I think would be a nice little addition and I don't see any major downsides. Anything I'm missing? -n [1] There's a lot more detail on this here: https://vorpus.org/blog/control-c-handling-in-python-and-trio/ but it's probably unnecessary for the current discussion. -- Nathaniel J. Smith -- https://vorpus.org

2 1

Add an option for delimiters in bytes.hex()
by robert.hoelzl＠posteo.de 04 May '17

04 May '17

The bytes.hex() function is the inverse function of Bytes.fromhex(). But fromhex can process spaces (which is much more readable), while hex() provides no way to include spaces. My proposal would be to add an optional delimiter, that allows to specify a string that will be inserted between the digit pairs of a byte: def hex(self, delimiter=‘‘): … This would allow to write: assert b’abc‘.hex(‘ ‘) == ’61 62 63‘ Gesendet von Mail für Windows 10

13 30

Suggestion: Add shutil.get_dir_size
by Ram Rachum 03 May '17

03 May '17

Hi, I have a suggestion: Add a function shutil.get_dir_size that gets the size of a directory, including all the items inside it recursively. I currently need this functionality and it looks like I'll have to write my own function for it. Cheers, Ram.

4 5

Augmented assignment syntax for objects.
by Erik 02 May '17

02 May '17

Hi. I suspect that this may have been discussed to death at some point in the past, but I've done some searching and I didn't come up with much. Apologies if I'm rehashing an old argument ;) I often find myself writing __init__ methods of the form: def __init__(self, foo, bar, baz, spam, ham): self.foo = foo self.bar = bar self.baz = baz self.spam = spam self.ham = ham This seems a little wordy and uses a lot of vertical space on the screen. Occasionally, I have considered something like: def __init__(self, foo, bar, baz, spam, ham): self.foo, self.bar, self.baz, self.spam, self.ham = \ foo, bar, baz, spam, ham ... just to make it a bit more compact - though in practice, I'd probably not do that with a list quite that long ... two or three items at most: def __init__(self, foo, bar, baz): self.foo, self.bar, self.baz = foo, bar, baz When I do that I'm torn because I know it has a runtime impact to create and unpack the implicit tuples and I'm also introducing a style asymmetry in my code just because of the number of parameters a method happens to have. So why not have an augmented assignment operator for object attributes? It addresses one of the same broad issues that the other augmented assignment operators were introduced for (that of repeatedly spelling names). The suggestion therefore is: def __init__(self, foo, bar, baz, spam, ham): self .= foo, bar, baz, spam, ham This is purely syntactic sugar for the original example: def __init__(self, foo, bar, baz, spam, ham): self.foo = foo self.bar = bar self.baz = baz self.spam = spam self.ham = ham ... so if any of the attributes have setters, then they are called as usual. It's purely a syntactic shorthand. Any token which is not suitable on the RHS of the dot in a standard "obj.attr =" assignment is a syntax error (no "self .= 1"). The comma-separators in the example are not creating a tuple object, they would work at the same level in the parser as the import statement's comma-separated lists - in the same way that "from pkg import a, b, c" is the same as saying: import pkg a = pkg.a b = pkg.b c = pkg.c ... "self .= a, b, c" is the same as writing: self.a = a self.b = b self.c = c E.

25 78

Proposed PEP 484 addition: describe a way of annotating decorated declarations
by Guido van Rossum 01 May '17

01 May '17

There's a PR to the peps proposal here: https://github.com/python/peps/pull/242 The full text of the current proposal is below. The motivation for this is that for complex decorators, even if the type checker can figure out what's going on (by taking the signature of the decorator into account), it's sometimes helpful to the human reader of the code to be reminded of the type after applying the decorators (or a stack thereof). Much discussion can be found in the PR. Note that we ended up having `Callable` in the type because there's no rule that says a decorator returns a function type (e.g. `property` doesn't). This is a small thing but I'd like to run it by a larger audience than the core mypy devs who have commented so far. Here's the proposed text (wordsmithing suggestions in the PR please): +Decorators +---------- + +Decorators can modify the types of the functions or classes they +decorate. Use the ``decorated_type`` decorator to declare the type of +the resulting item after all other decorators have been applied:: + + from typing import ContextManager, Iterator, decorated_type + from contextlib import contextmanager + + class DatabaseSession: ... + + @decorated_type(Callable[[str], ContextManager[DatabaseSession]]) + @contextmanager + def session(url: str) -> Iterator[DatabaseSession]: + s = DatabaseSession(url) + try: + yield s + finally: + s.close() + +The argument of ``decorated_type`` is a type annotation on the name +being declared (``session``, in the example above). If you have +multiple decorators, ``decorated_type`` must be topmost. The +``decorated_type`` decorator is invalid on a function declaration that +is also decorated with ``overload``, but you can annotate the +implementation of the overload series with ``decorated_type``. + -- --Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>)

2 2

Decorators for running a function in a Process or Thread
by NoxDaFox 01 May '17

01 May '17

The decorators would abstract the logic of spawning a process or thread and maintaining its lifecycle. I think it could be a good fit for the `concurrent.futures` module. Decorated functions would return a `Future` object and run the logic in a separate thread or process. @concurrent.futures.thread def function(arg, kwarg=0): return arg + kwarg future = function(1, kwarg=2) future.result() The Process decorator in particular would support use cases such as running unstable code within an application taking advantage of process separation benefits. I often found myself relying on external APIs which would either crash or run indefinitely affecting my application stability and performance. @concurrent.futures.process(timeout=60): def unstable_function(arg, kwarg=0): # hang or segfault here return arg + kwarg future = unstable_function(1, kwarg=2) try: future.result() except TimeoutError as error: print("Function took more than %d seconds" % error.args[1]) except ProcessExpired as error: print("Process exit code %d" % error.exitcode) In case of timeout, the process would be terminated reclaiming back its resources. Few years ago I wrote a library for this purpose which turned out pretty handy in reducing the boilerplate code when dealing with threads and processes. It could be a starting point for discussing about the implementation. https://pypi.python.org/pypi/Pebble

3 2