Mailman 3 August 2019 - Python-ideas

Re: Namespace context managers
by Niklas Rosenstein 19 Aug '19

19 Aug '19

You may be interested in this working example of a @namespace decorator (for functions though, not for classes). https://stackoverflow.com/a/52358426/791713 — @namespace def mynamespace(): eggs = 'spam' class Bar: def hello(self): print("Hello, World!") assert mynamespace.eggs == 'spam' mynamespace.Bar().hello() — It uses sys.settrace() to find the frame of the function before it is called to convert it into an actual module object. My (largely underdeveloped and not recommended to be used) “Node.py” project implements a preprocessor for Python code in order to implement such a namespace keyword. — namespace Example: def hello(name): print('Hello, {}!'.format(name)) Example.hello('World') — It does this by converting the code above to the below (and _NamespaceSyntax__namespace_decorator points to the same namespace decorator from above) — @require._NamespaceSyntax__namespace_decorator def Example(): def hello(name): print('Hello, {}!'.format(name)) — (Link to code here: https://github.com/nodepy/nodepy/blob/f9e572aa20159af7a1a57b503d0c57693c7a9… ) Personally I like the suggestion by Ricky Teachey to use “def” instead of introducing a new keyword. If we cannot settle on this kind of syntax addition, the namespace decorator may be a good addition to the standard library? The full code below for reference. — import sys import types def call_function_get_frame(func, *args, **kwargs): """ Calls the function *func* with the specified arguments and keyword arguments and snatches its local frame before it actually executes. """ frame = None trace = sys.gettrace() def snatch_locals(_frame, name, arg): nonlocal frame if frame is None and name == 'call': frame = _frame sys.settrace(trace) return trace sys.settrace(snatch_locals) try: result = func(*args, **kwargs) finally: sys.settrace(trace) return frame, result def namespace(func): frame, result = call_function_get_frame(func) try: module = types.ModuleType(func.__name__) module.__dict__.update(frame.f_locals) return module finally: del frame — Best, -Niklas

2 1

Extend ast with types for <type>* instead of using raw lists
by Caleb Donovick 17 Aug '19

17 Aug '19

When walking an ast it impossible to know the type of an empty list without writing down some giant lookup from node types and field names to field types. More concretely it would nice be to able to programatically visit all blocks (stmt*) without having to something like: ``` class BlockVisitor(NodeVisitor): def visit_If(self, node: If): self.visit(node.test) self.visit_block(node.body) self.visit_block(node.orelse) def visit_FunctionDef(self, node: FunctionDef): for field, value in iter_fields(node): if field == 'body': self.visit_block(value) else: # the implementation of generic_visit ``` Now it turns out that all fields that are lists and are named "body", "orelse", or "finalbody" are stmt* and only such fields are stmt*. A rule could also be synthesized to identify expr* and so forth but this seems incredibly hacky to me. It would be much cleaner if <type>* were actual nodes in the ast. E.g. something like: ``` class ast_list(AST, MutableSequence[T_co]): ... class StmtList(ast_list[stmt]): ... class ExprList(ast_list[expr]): ... ... class FunctionDef(stmt): name: identifier args: arguments body: StmtList decorator_list: ExprList returns: Optional[expr] ``` This would not change the behavior or structure in any way other than tagging <type>* and allowing <type>* to be visited. It would potentially break old code which relies on stuff like `if isinstance(node.field, list)` e.g. the implementation of generic_visit. Caleb Donovick

4 6

Add a `get_profile_dict` function to cProfile
by olshansky.daniel＠gmail.com 17 Aug '19

17 Aug '19

Hey everyone, I've been using cProfile to profile some of my code, but found out that while the tools to print the results are very convenient, it was difficult to extract and send to a logging tool (i.e. fluentd) where it can be further parsed and analyzed. I'm thinking of adding a function to cProfile (https://github.com/python/cpython/blob/master/Lib/cProfile.py) similar to "get_profile_dict" below, but wanted to get some feedback from the community before I submit a proper PR for it. The code below is just a sample of how I used the proposed function to extract the analysis of a Fibonacci call. Thanks! ``` import cProfile, pstats import pprint from pstats import func_std_string, f8 def fib(n): if n == 0: return 0 if n == 1: return 1 return fib(n-1) + fib(n-2) pr = cProfile.Profile() pr.enable() fib(5) pr.create_stats() ps = pstats.Stats(pr).sort_stats('tottime', 'cumtime') def get_profile_dict(ps): profile_dict = {} width, list = ps.get_print_list([]) if list: for func in list: cc, nc, tt, ct, callers = ps.stats[func] ncalls = str(nc) if nc == cc else (str(nc) + '/' + str(cc)) tottime = float(f8(tt)) percall_tottime = -1 if nc == 0 else float(f8(tt/nc)) cumtime = float(f8(ct)) percall_cumtime = -1 if cc == 0 else float(f8(ct/cc)) func_name = func_std_string(func) profile_dict[func_name] = { "ncalls": ncalls, "tottime": tottime, # time spent in this function alone "percall_tottime": percall_tottime, "cumtime": cumtime, # time spent in the function plus all functions that this function called, "percall_cumtime": percall_cumtime } profile_dict["total_tt"] = float(f8(ps.total_tt)) return profile_dict pp = pprint.PrettyPrinter(depth=6) pp.pprint(get_profile_dict(ps)) ```

1 0

Exception handling in objects
by Marat Sharafutdinov 15 Aug '19

15 Aug '19

While using any library there is a need to handle exceptions it raises. In ideal conditions author of library defines base exception and builds hierarchy of exceptions based on it. Moreover, all exceptions a library raises (including those that do not inherit its base exception) are documented and this information is kept up to date. Unfortunately, this is not always the case. Library authors are free to use the language exception mechanism in any way. So instead of using documented features of library as a black box, the user often has to plunge into the source code of a library itself and look for exceptions it could raise in order to be prepared for them. The essence of my idea is simple and does not affects compatibility with existing code: to automatically define a magic attribute of module under the working name "__exception__" while importing any module. It can be used in usual way, for example: >>> try: ... any_module.anything ... except any_module.__ exception__: ... ... Here any exception an author of library raises can be handled. This way you can handle any exception that occurs in the library without even knowing what exceptions it can raise. The important point is that only exceptions raised in library itself are handled. Those exceptions that can be raised in underlying libraries and so on through the dependency chain should be handled separately. Developing this idea, it could be more universal mechanism that works not only with modules, but with any objects of the language. For instance: >>> try: ... any_function() ... except any_function.__ exception__: ... ... or >>> try: ... AnyClass.anything ... except AnyClass.__ exception__: ... ... It is important to understand the key difference between this mechanism and the capture of broad exception: only those exceptions are handled that are RAISED EXPLICITLY inside the object, while broad exception captures ANY exceptions that may occur in the object. It seems to me this idea will help library users more correctly handle their exceptions if this is problematic. This will save time and make code more reliable.

6 6

hybrid implementation for PyLongObject (performance)
by malincns＠163.com 14 Aug '19

14 Aug '19

The idea is mixing `PyLongObject` with `Python 2's PyIntObject` implementation. For example, on a 64-bit platform, if (an integer >=-9223372036854775808 and <=9223372036854775807), PyLongObject uses a native C type `signed long` to represent it. People mostly use +-* operations, maybe using native int is faster, even including the cost of overflow check. If operation will overflow or other operations like **, we can transform native int to current form, and run in current code path. Regards

4 4

Stdlib module - Draft
by Abdur-Rahmaan Janhangeer 13 Aug '19

13 Aug '19

Greetings, Here is a draft after the stdlib module discussion <https://mail.python.org/archives/list/python-ideas@python.org/thread/XSYEVR…> . Abstract ======= Proposal: Adding a stdlib module to the standard library Defining the stdlib ============== Would the stdlib module be OS-independent or not? A better approach would be to go along the standard modules as defined in the docs <https://docs.python.org/3/library/index.html>. You get to know all modules within the stdlib, it even lists curses on windows (see benefit at 2. below) with info. But it does not let you access it if you want to use it when importing. Same for the third issue: The third issue is alternative Python implementations, discussed at concerns no.2. Advantages of having a stdlib module ============================= Below are some benefits of having such a module: - 1 Knowing what module can be imported without installing This can be useful for interactive learning. Like with the inbuilt builtins library, one get to know what builtins are there in Python, though it's not the lib's only use. In the spirit of the keyword module, it can also check if an import forms part of the std lib - 2 Check for accidental name collisions It might come in handy to check for python platform collisions. An intersection between their modules will do the job - 3 Namespaced imports import math could be used as stdlib.math to better distinguish. Concerns ======== - 1. Backward compatibility Python 4.0 codes might contain codes incompatible with previous versions - 2. Different standard modules If a Python implementation adds in a module, would codes implementing stdlib.fancy_lib on a diffrent platform not get broken? That is essentially the same as executing curses on windows or RPi on Raspberry Pi - Raspian. Just like RPi codes would not run on a windows, similarly, stdlib.fancy_lib would not run on the standard C implementation. Ideas and Implementations ==================== - Andrew Barnert implemented a version of stdlib <https://github.com/abarnert/stdlib> where you can import without having to imply where the module is meaning instead of from x.y import z, you do: from stdlib import z - Each implementation could hardcode the names of std modules which could be accessed by stdlib for listing and imports - Use what help('modules') uses i.e pkgutil.walk_packages minus 3rd party modules - This stdlib implemenation <https://github.com/jackmaney/python-stdlib-list> from Jack Maney essentially hardcodes the libs like here <https://github.com/jackmaney/python-stdlib-list/blob/master/stdlib_list/lis…> . - The real implementation is up to ... be implemented. Draft containing inputs from: Andrew Barnert Chris Angelico Christopher Barker Steve Barnes Steven D'Aprano -- Abdur-Rahmaan Janhangeer pythonmembers.club <http://www.pythonmembers.club/> | github.com <https://github.com/Abdur-rahmaanJ> Mauritius

2 2

Re: Idiomatic API for shell scripting
by Greg Price 13 Aug '19

13 Aug '19

(Changed subject, as I don't think a PEP is the right focus for thinking about this problem space) Thiago Padilha wrote: > Python is often used a shell scripting language due to it being more > robust than traditional Unix shells for managing complex programs, but > it is significantly more verbose than Bash & friends for doing common > shell scripting tasks such as writing pipelines and redirection . I 100% agree with this problem statement. As others have mentioned, there are several libraries people have written that try to do something in this direction. I’ve never found one that I felt approached the problem from quite the right angle, so I’ve been experimenting recently with my own solution (incorporating ideas from friends.) I guess this is a good prompt to discuss it more widely! All the ideas below are in the working implementation at https://github.com/gnprice/pysh . ---------- First, one unsung strength of Bash (and friends) that I think is actually an essential foundation is what happens when there’s no redirection, no pipelines, no fancy substitutions — you want to just run a program, and pass it some arguments. The core thing you need to do here is make a list of strings, for the command and its arguments. That’s how the underlying API works. So for example in Bash: # 5 strings: gpg -d -o "$cleartext_path" "$cryptotext_path" # (4 + len(commit_ids)) strings: git merge "${commit_ids[@]}" -m "$message" In Python this typically ends up looking like: subprocess.check_call(['gpg', '-d', '-o', cleartext_path, cryptotext_path]) subprocess.check_call(['git', 'merge'] + commit_ids + ['-m', message]) For the arguments that are literals — which often means most of them -- we keep saying quote-comma-space-quote, quote-comma-space-quote, effectively as a delimiter. That’s a lot of visual noise, as well as extra typing. And when you want to splice in a whole list of arguments, it gets worse. What we want here is something almost like this: subprocess.check_call( f'gpg -d -o {cleartext_path} {cryptotext_path}'.split()) # BAD! ... except that version steps into the classic shell pitfall where if a value happens to contain whitespace, it turns into several arguments and totally changes the meaning. But it’s so close. Really we just want to do that but with the `split` first — so it operates on the literal string you see in the source code — and *then* the `format`, or the f-string substitution. In fact you can implement a 70% solution in just a line: def shwords(fmt, **kwargs): return [word.format(**kwargs) for word in fmt.split()] >>> shwords('rm -rf {tmpdir}/{userdoc}', tmpdir='/tmp', userdoc='1 .. 2') ['rm', '-rf', '/tmp/1 .. 2'] A bit more work gives you f-string-like behavior, if you opt into it: subprocess.check_call(shwords_f('rm -rf {tmpdir}/{userdoc}')) check_cmd_f('rm -rf {tmpdir}/{userdoc}') # small convenience helper and some more gives you positional arguments: check_cmd('gpg -d -o {} {}', cleartext_path, cryptotext_path) and a format-minilanguage extension `{...!@}` , to substitute in a whole list: check_cmd('git merge {!@} -m {}', commit_ids, message) Full implementation here: https://github.com/gnprice/pysh/blob/8faa55b06/pysh/words.py As far as I know, this idea is a new one. I’ve found it goes a long way in making scripts that run a lot of external commands feel Pythonically concise and low-boilerplate. ---------- Then there are a number of features on top of that foundation that we need to really make interacting with external Unix-style commands as convenient as it can be. To avoid making this email longer I’ll just briefly gesture at two of them for now — more details in the repo! * Getting a command’s output in convenient form — not only on stdout, but success/failure from the return code: if not try_cmd('git diff-index --quiet HEAD'): raise ClickException("worktree not clean") # ... commit_id = try_slurp_cmd_f('git rev-parse --verify --quiet {name}') if commit_id is None: commit_id = try_slurp_cmd_f('git rev-parse --verify --quiet origin/{name}') if commit_id is None: raise MissingBranchException(name) * Pipelines. :-) The `|` operator really is too good to pass up: def first_mention(pattern): return pysh.slurp( cmd.run('git log --oneline --reverse --abbrev=9') | cmd.run('grep -m1 {}', pattern) | cmd.run('perl -lane {}', 'print $F[0]') ) I think one important direction for pipelines is making it convenient to stick bits of Python code in the middle of the pipeline, in amongst external programs. That direction isn’t fully developed in the current API. ---------- For anyone who finds this interesting, I’d encourage you to check out the demo scripts: https://github.com/gnprice/pysh/tree/master/example Then if you take some interesting fragment of script you have handy and try converting it to use this library, I’d be very curious to hear how it turns out! That’s what the demo/example scripts are for, and I think it’s the main ingredient this work needs at this stage: seeing how the API works out in a range of use cases, and seeing what patterns it can be improved to better serve. Also naturally I will be glad to discuss it here for as long as others are interested. Cheers, Greg

1 0

Disallow all ending statements (continue/break/return) inside of finally
by Batuhan Taskaya 13 Aug '19

13 Aug '19

About 2 days ago i was writing a blog post and i thought why dont i use a

1 1

Exceptions with Message Templates
by Ryan Fox 13 Aug '19

13 Aug '19

Exception definitions in Python are not great. There are two main ways they're used in the code that I've read: 1) By far the most common: >>> class MyException(Exception): ... pass >>> raise MyException(f'Bad thing happened during {action} in {context}') 2) Much rarer: >>> class MyException(Exception): ... def __init__(self, action, context): ... super().__init__(f'Bad thing happened during {action} in {context}') >>> raise MyException(current_action, current_context) Version 1 is quick and easy, but messages skew and become inconsistent as they're copied from place to place. The Python standard library isn't immune to this either. Version 2 looks simple enough to do, but all of the repetitious boilerplate adds up when several exception types need to be defined. (And it's even worse when you want all of your code to include typing annotations.) Most people don't bother. My proposal is a new exception class as the preferred base for user-defined exceptions: >>> class MyException(ExceptionTemplate): ... message = 'Bad thing happened during {action} in {context}' >>> raise MyException(action=current_action, context=current_context) I have a reference implementation, implemented in an almost trivial amount of pure Python code: https://github.com/rcfox/exception-template/blob/master/exception_template/… So why am I bothering you with this? I would like to see this become the preferred method of defining new exceptions as recommended by the Python documentation. This would also require adding ExceptionTemplate as a built-in exception type. I will grant that ExceptionTemplate seems a little bit magical, but all of the required arguments are explicitly laid out to the user in the 'message' field, and format strings should be a familiar concept to most users. I've also included some tests that show that you can still treat these exceptions as regular classes: https://github.com/rcfox/exception-template/blob/master/tests/test.py#L53

5 12

Namespace context managers
by Batuhan Taskaya 10 Aug '19

10 Aug '19

I am proposing namespace context managers with implementing `__enter__` and `__exit__` on dict objects. It would make closures possible in python with a pythonic syntax. a = 4 namespace = {} with namespace: a = 3 assert a == 4 assert namespace["a"] == 3

15 32