Mailman 3 September 2017 - Python-ideas

allow overriding files used for the input builtin
by Wren Turkal 29 Sep '17

29 Sep '17

Hi there, I have posted an idea for improvement with a PR of an implementation to https://bugs.python.org/issue31603. The basic idea is to add fin, fout, and ferr file object parameters and default to using what is used today when the args are not specified. I believe this would be useful to allow captures input and send output to specific files when using input. The input builtin has some logic to use readline if it's available. It would be nice to be able to use this same logic no matter what files are being used for input/output. This is meant to turn code like the following: orig_stdin = sys.stdin orig_stdout = sys.stdout with open('/dev/tty', 'r+') as f: sys.stdin = f sys.stdout = f name = input('Name? ') sys.stdin = orig_stdin sys.stdout = orig_stdout print(name) into something more like this: with open('/dev/tty', 'r+') as f: name = input('Name? ', fin=f, fout=f) print(name) It's nice that it makes the assignment to a global variable to change the file used for input/output to no longer be needed. I had this idea the other day, and I realized that it would be super easy to implement, so I went ahead the threw up a PR also. Would love to see if anyone else is interested in this. I think it's pretty cool that the core logic really didn't need to be changed other than plumbing in the new args. FWIW, this change introduces no regressions and adds a few more tests to test the new functionality. Honestly, I think this functionality could probably be used to simplify some of the other tests as well, but I wanted to gauge what folks thought of the change before going farther. Wren Turkal Existential Production Engineer of the Ages Facebook, Inc.

5 12

Re: [Python-ideas] allow overriding files used for the input builtin
by Adam Bartoš 29 Sep '17

29 Sep '17

Hello, it seems that there are two independent issues – a way to temporarily replace all sys.std* streams, and a way to use the special interactive readline logic for arbitrary terminal-like file. I thought that OP's concern was the latter. In that case shouldn't there just be a way to produce an IO object f whose .readline() (or other special method) does this and has an optional prompt argument? The OP's code would become with open_terminal('/dev/tty') as f: name = f.readline(prompt='Name? ') It seems to me that changing the rather specialized input() function is wrong in both cases. For me input() is just a short way to read from the standard input, usually interactively – the same way print is a short way to write to the standard output. IIRC there was even proposal to remove input() in favor of direct use of sys.stdin.readline() at time of Python 3000 redesign. Regards, Adam Bartoš

1 0

Fwd: A PEP to define basical metric which allows to guarantee minimal code quality
by Pavol Lisy 25 Sep '17

25 Sep '17

On 9/20/17, alexandre.galode(a)gmail.com <alexandre.galode(a)gmail.com> wrote: [...] > But i think, as wee need to avoid to talk about any tool name in the PEP, > we need to avoid to give a code example. The aim of this proposal is to > have a guideline on minimal metrics to have minimal quality. Michel Foucault wrote book (https://en.wikipedia.org/wiki/The_Birth_of_the_Clinic) which subtitle is "An Archaeology of Medical Perception". Imagine that code is organism and quality is health. Which archaeological era of medical perception is analogous to your proposal? ( https://en.wikipedia.org/wiki/File:Yupik_shaman_Nushagak.jpg or https://en.wikipedia.org/wiki/Blinded_experiment ?) PS. Good analogy is probably heart rate. It is very simple metric which we could use. And it seems not too problematic to propose range where it is healthy. But different kind of software is like different species. (see for example http://www.merckvetmanual.com/appendixes/reference-guides/resting-heart-rat… ) And there are development stages! Embryonic bpm is NA (because no heart) and then much bigger than child's and its bigger than adult's. Another simple metric could be temperature. But individual has also different tissues! Different part of body have different temperature. (We could probably not measure hair's temperature) etc, etc...

3 2

Hexadecimal floating literals
by Serhiy Storchaka 24 Sep '17

24 Sep '17

The support of hexadecimal floating literals (like 0xC.68p+2) is included in just released C++17 standard. Seems this becomes a mainstream. In Python float.hex() returns hexadecimal string representation. Is it a time to add more support of hexadecimal floating literals? Accept them in float constructor and in Python parser? And maybe add support of hexadecimal formatting ('%x' and '{:x}')?

23 46

A PEP to define basical metric which allows to guarantee minimal code quality
by Alexandre GALODE 21 Sep '17

21 Sep '17

Hi everybody, I'm a Python dev since about 10 years. I saw that customers are more and more curious about development process, talking from various reference on which allow to define what they call "good quality code". Someones are talking only about Pylint, other one from SonarQube, ... I search for SonarQube for Python, and saw it was using Pylint, coverage and Unittest. I saw also some customers internal tools which, every of them, was using internal tool, pylint, pylama, pep8, ... So, with help of my customers (thanks to them ^^), i realize there was a missing in PEP about this point. I made some search, and saw that there is no PEP which define, not even a minimum, which metrics and/or tools use to evaluate the quality of a Python code; no minimal indication in this goal. Also, my PEP idea was to try to define basical metrics to guarantee minimal code quality. I'd like to purpose it as an informational PEP. So users and implementers would be free to ignore it. I does not see it as a constrining PEP, but as a PEP reference which could give help, and be used by every developers to say he/she is PEP xxxx from its development. What are you thinking about this PEP idea?

11 17

Move some regrtest or test.support features into unittest?
by Victor Stinner 19 Sep '17

19 Sep '17

Hi, tl; dr How can we extend unittest module to plug new checks before/after running tests? The CPython project has a big test suite in the Lib/test/ directory. While all tests are written with the unittest module and the unittest.TestCase class, tests are not run directly by unittest, but run by "regrtest" (for "regression test") which is a test runner doing more checks (and more). I would like to see if and how we can integrate/move some regrtest features into the unittest module. Example of regrtest features: * skip a test if it allocates too much memory, command line argument to specify how many memory a test is allowed to allocate (ex: --memlimit=2G for 2 GB of memory) * concept of "resource" like "network" (connect to external network servers, to the Internet), "cpu" (CPU intensive tests), etc. Tests are skipped by default and enabled by the -u command line option (ex: "-u cpu). * track memory leaks: check the reference counter, check the number of allocated memory blocks, check the number of open file descriptors. * detect if the test spawned a thread or process and the thread/process is still running at the test exit * --timeout: watchdog killing the test if the run time exceed the timeout in seconds (use faulthandler.dump_traceback_later) * multiprocessing: run tests in subprocesses, in parallel * redirect stdout/stderr to pipes (StringIO objects), ignore them on success, or dump them to stdout/stderr on test failure * --slowest: top 10 of the slowest tests * --randomize: randomize test order * --match, --matchfile, -x: filter tests * --forever: run the test in a loop until it fails (or is interrupted by CTRL+c) * --list-tests / --list-cases: list test files / test methods * --fail-env-changed: mark tests as failed if a test altered the environment * detect if a "global variable" of the standard library was modified but not restored by the test: resources = ('sys.argv', 'cwd', 'sys.stdin', 'sys.stdout', 'sys.stderr', 'os.environ', 'sys.path', 'sys.path_hooks', '__import__', 'warnings.filters', 'asyncore.socket_map', 'logging._handlers', 'logging._handlerList', 'sys.gettrace', 'sys.warnoptions', 'multiprocessing.process._dangling', 'threading._dangling', 'sysconfig._CONFIG_VARS', 'sysconfig._INSTALL_SCHEMES', 'files', 'locale', 'warnings.showwarning', 'shutil_archive_formats', 'shutil_unpack_formats', ) * test.bisect: bisection to identify the failing method, used to track memory leaks or identify a test leaking a resource (ex: create a file but don't remove it) * ... : regrtest has many many features My question is also connected to test.support (Lib/test/support/__init__.py): a big module containing a lot of helper functions to write tests and to detect bugs in tests. For example, @reap_children decorator emits a warnig if the test leaks a child process (and reads its exit status to prevent zombie process). I started to duplicate code in many files of Lib/test/test_*.py to check if tests "leak running threads" ("dangling threads"). Example from Lib/test/test_theading.py: class BaseTestCase(unittest.TestCase): def setUp(self): self._threads = test.support.threading_setup() def tearDown(self): test.support.threading_cleanup(*self._threads) test.support.reap_children() I would like to get this test "for free" directly from the regular unittest.TestCase class, but I don't know how to extend the unittest module for that? Victor

4 6

Use __all__ for dir(module) (Was: PEP 562)
by Cody Piersall 18 Sep '17

18 Sep '17

On Tue, Sep 12, 2017 at 3:26 AM, Ivan Levkivskyi <levkivskyi(a)gmail.com> wrote: > @Cody >> I still think the better way >> to solve the custom dir() would be to change the module __dir__ >> method to check if __all__ is defined and use it to generate the >> result if it exists. This seems like a logical enhancement to me, >> and I'm planning on writing a patch to implement this. Whether it >> would be accepted is still an open issue though. > > This seems a reasonable rule to me, I can also make this patch if > you will not have time. I submitted a PR:https://github.com/python/cpython/pull/3610 and a BPO issue: https://bugs.python.org/issue31503 R. David Murray pointed out that this is a backwards-incompatible change. This is technically true, but I don't know of any code that depends on this behavior. (Of course, that does not mean it does not exist!) >From my perspective, the big benefit of this change is that tab-completion will get better for libraries which are already defining __all__. This will make for a better REPL experience. The only code in the stdlib that broke were tests in test_pkg which were explicitly checking the return value of dir(). Apart from that, nothing broke. If a module does not have __all__ defined, then nothing changes for that module. Cody

3 3

Make map() better
by Jason H 17 Sep '17

17 Sep '17

The format of map seems off. Coming from JS, all the functions come second. I think this approach is superior. Currently: map(lambda x: chr(ord('a')+x), range(26)) # ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'] But I think this reads better: map(range(26), lambda x: chr(ord('a')+x)) Currently that results in: TypeError: argument 2 to map() must support iteration Also, how are we to tell what supports map()? Any iterable should be able to map via: range(26).map(lambda x: chr(ord('a')+x))) While the line length is the same, I think the latter is much more readable, and the member method avoids parameter order confusion For the global map(), having the iterable first also increases reliability because the lambda function is highly variable in length, where as parameter names are generally shorter than even the longest lambda expression. More readable: IMHO: map(in, lambda x: chr(ord('a')+x)) out = map(out, lambda x: chr(ord('a')+x)) out = map(out, lambda x: chr(ord('a')+x)) Less readable (I have to parse the lambda): map(lambda x: chr(ord('a')+x), in) out = map(lambda x: chr(ord('a')+x), out) out = map(lambda x: chr(ord('a')+x), out) But I contend: range(26).map(lambda x: chr(ord('a')+x))) is superior to all.

15 25

sys.py
by Eric Snow 14 Sep '17

14 Sep '17

The sys module is a rather special case as far as modules go. It is effectively a "console" into the interpreter's internal state and that includes some mutable state. Since it is a module, we don't have much of an opportunity to: * validate values assigned to its attributes [1] * issue DeprecationWarning for deprecated attrs [2] * alias attrs [2] * replace get (and get/set) functions with properties * re-organize sys [3] One possible solution I've been toying with for quite a while [2] is to rename the current sys module "_sys" and then add Lib/sys.py. The new module would proxy the old one (for backward-compatibility), but also allow us to do all of the above (e.g. validate sys.modules). I implemented this a few weeks ago: https://github.com/ericsnowcurrently/cpython/tree/sys-module (for the sake of comparison: https://github.com/ericsnowcurrently/cpython/pull/2) It uses the trick of replacing itself in sys.modules (though it could just as well set __class__). The only problem I've encountered is code that uses "type(sys)" to get types.ModuleType. Under my branch "type(sys)" returns the ModuleType subclass. Otherwise everything looks fine. Thoughts? -eric [1] I ran into this recently: https://bugs.python.org/issue31404 [2] I have plans for a low-level encapsulation of the import state and a high-level API for the current import machinery as a whole. As part of that I'd like to be able to deprecate the current import state attrs (e.g. sys.modules) and make them look up values from sys.importstate. [3] This has come up before, including https://www.python.org/dev/peps/pep-3139/. Also PEP 432 implies a clearer structure for sys.

4 6

PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code
by Eric Snow 14 Sep '17

14 Sep '17

Hi all, As part of the multi-core work I'm proposing the addition of the "interpreters" module to the stdlib. This will expose the existing subinterpreters C-API to Python code. I've purposefully kept the API simple. Please let me know what you think. -eric https://www.python.org/dev/peps/pep-0554/ https://github.com/python/peps/blob/master/pep-0554.rst https://github.com/python/cpython/pull/1748 https://github.com/python/cpython/pull/1802 https://github.com/ericsnowcurrently/cpython/tree/high-level-interpreters-m… ********************************************** PEP: 554 Title: Multiple Interpreters in the Stdlib Author: Eric Snow <ericsnowcurrently(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2017-09-05 Python-Version: 3.7 Post-History: Abstract ======== This proposal introduces the stdlib "interpreters" module. It exposes the basic functionality of subinterpreters that exists in the C-API. Rationale ========= Running code in multiple interpreters provides a useful level of isolation within the same process. This can be leveraged in number of ways. Furthermore, subinterpreters provide a well-defined framework in which such isolation may extended. CPython has supported subinterpreters, with increasing levels of support, since version 1.5. While the feature has the potential to be a powerful tool, subinterpreters have suffered from neglect because they are not available directly from Python. Exposing the existing functionality in the stdlib will help reverse the situation. Proposal ======== The "interpreters" module will be added to the stdlib. It will provide a high-level interface to subinterpreters and wrap the low-level "_interpreters" module. The proposed API is inspired by the threading module. The module provides the following functions: enumerate(): Return a list of all existing interpreters. get_current(): Return the currently running interpreter. get_main(): Return the main interpreter. create(): Initialize a new Python interpreter and return it. The interpreter will be created in the current thread and will remain idle until something is run in it. The module also provides the following class: Interpreter(id): id: The interpreter's ID (read-only). is_running(): Return whether or not the interpreter is currently running. destroy(): Finalize and destroy the interpreter. run(code): Run the provided Python code in the interpreter, in the current OS thread. Supported code: source text. Copyright ========= This document has been placed in the public domain.

11 36