Python-ideas July 2020

python-ideas@python.org

93 participants
54 discussions

by s.williamswynn.mail＠gmail.com

Evening, I have run into a situations when an enum member becomes obsolete. However, I'd like to avoid removing the member from the enum to give time for users to catch up to a libraries core types. I've seen Java has a @Deprecated annotation for their Enums. Python enum's can also set an _ignore_ attribute which will avoid instantiating a member in the list. I've implemented a similar approach to solve deprecating enum members. You can check it out here: https://github.com/foxyblue/cpython/pull/1

4 months

Experimenting with dict performance, and an immutable dict

by Marco Sulla

Let me first say that the code and discussion, as per title, is also about possible performance improvements to the dict base type. TL;DR I implemented a frozendict using CPython 3.9 code. It seems that an immutable dict *could* be faster than dict in some cases. Furthermore, some optimization could be applied to dict too. Long explaining: Since now I have some time, I decided to experiment a little with the creation of an immutable dict in Python. Unluckily, I started this experiment many months ago, so the CPython code I used is old. Maybe some or all of my considerations are outdated. Initially, I wrote a quick and dirty implementation: https://github.com/Marco-Sulla/cpython/commit/fde4e6d236b19636063f8afedea8c… The code was very simple, and the performance was identical to dict. So in theory, adding a frozendict to CPython is not hard. But there's more. After the first implementation, I started to try to improve the performance of frozendict. The result of the improvements are here: https://github.com/Marco-Sulla/cpython/blob/master/frozendict/test/bench.txt For benchmarks, I used simply timeit, with autorange and repeat and, as suggested in the module documentation, I got the minimum of the results. Here is the code: https://github.com/Marco-Sulla/cpython/blob/master/frozendict/test/bench.py I have not tested with an optimized build, since optimization data is collected using the unit tests, and I didn't write tests for frozendict in the official CPython test framework. The tests and benchmarks were done using CPython 3.9a0. CPU and other pc resources were not restricted using pyperf or similar tools, to see the "real" speed. CPython was compiled using gcc and g++. In benchmarks, I compared methods and operators using dict and frozendict. The benchmark uses dicts with all integers and dicts with all strings. Furthermore, I tested dicts with size 8 (the most optimized size in CPython) and 1000 elements (maybe too much, but I wanted to see how they perform with a high number of elements). Every benchmark has a line in the output. The Name describes the benchmarked method, operator or code snippet. Size is the size of the dict, 8 or 1000. Keys are the keys type, str or int. Type is the dictionary type, dict or frozendict. In Name, the "o" represents the object itself (dict or frozendict). "d", in benchmark with dict, is "o"; in benchmarks with frozendict is an equivalent instance of type dict. Some consideration: 1. frozendict is very fast, as expected, at copying. But it's also faster at creation, using a (not frozen) dict, kwargs or a sequence2. Speedups range from 20% to 45%. 2. frozendict is also a bit faster when you iterate over it, especially over values, where is ~15% faster 3. hash seems really fast, but this is because it's cached the first time hash() is invoked 4. where frozendict is slower is when you unpickle it and with fromkeys and repr. This is because I wrote a very naif implementation of these methods, without optimizing them. The other methods have a comparable speed. Here is the code: https://github.com/Marco-Sulla/cpython Here is the diff between the CPython code and my commits: https://github.com/python/cpython/compare/master...Marco-Sulla:master About code I coded the implementation doing a simple copy/paste of the existing dict functions, modifying their code and renaming them. This way I'm sure dict continues to work as before, and I can compare the speed gain. Some of the optimization I adopted can be implemented in `dict` too. For example, instead of creating an empty dict and resizing it, I create it with the "maximum" size and I fill it. It seems to work, even if I did not explore the possibility that a mutable object can change while a frozendict creation is in progress. Some problems with optimizing dict and maintaining a frozendict: 1. duplication of code. To gain a significant boost, I had to copy and paste a lot of code. Functions can be remerged again, but maybe the speedup will be reduced. 2. split vs combined dicts. As I wrote, split dicts seem to be faster in reading than combined dicts. For example, iterating over values is faster with a split dict, as expected. But writing was not tested; furthermore, some of the optimizations can be adopted for dicts too, so the convenience of a split table can be lowered. dict continues to maintain both split and combined tables, so this could be not a problem. But the code could be less and more fast if only a table layout is supported 3. the CPython code I used is old, so some of the improvements I adopted could be already implemented About frozendict Apart the considerations done in the [PEP 416]( https://www.python.org/dev/peps/pep-0416/), that was rejected since there was little gain from its implementation, I think that frozendict can be useful as a substitute of MappingProxyType, that is really slow. MappingProxyType is not much used, but it's present in critical parts of CPython code, for example in _sre. We have to see if a mapping proxy type *can* be substituted with an immutable map in some critical part of CPython code. Furthermore, frozendicts could be used for implementing "immutable" classes and modules, and can be used as a faster dict if its content does not change.

4 months

Re: Access (ordered) dict by index; insert slice

by Christopher Barker

Is there ANY chance of setting the default reply-to to the list? I know everyone else thinks that's a bid idea, but I doubt this was actually intended just for me. If it was, I apologize for bringing it back on the list. On Sun, Jul 12, 2020 at 10:12 PM Inada Naoki <songofacandy(a)gmail.com> wrote: > On Mon, Jul 13, 2020 at 1:51 PM Christopher Barker <pythonchb(a)gmail.com> > wrote: > As I and Eric already said several times, we can do it in C already -- > itertools.islice. > > ... itertools.islice is implemented in C too. No need to call next() from > Python stack. > Ahh -- thanks! I had no idea islice() was in C -- yes, then, there probably is little to gain from a C implementation. I'd still be interested in the performance comparison, but not enough to try to write a direct-access-to-the-internals-of-dict C version myself :-) I still like the idea of indexing the order-preserving dict, but we do seem to not have any compelling use cases. Particularly if someone does get a reservoir sampling implementation in the stdlib. (see another thread for that) -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

4 months, 1 week

Simplifying functions syntax

by Thiago Carvalho D' Ávila

Python is a simple language to understand, but to me functions are a special case. And according to the Zen of Python (PEP20 <https://www.python.org/dev/peps/pep-0020/>): `Special cases aren't special enough to break the rules.`. For beginners, Python functions can cause some pain that boils down to three points that shows some lack of standardization. 1. Function declaration needs keyword `def`: If you are a long time Python (1991) developer, you surely got used to the keyword `def` to declare a function, same goes if you are a Ruby (1995) programmer. In Fortran (1957) you start them with the word `function` and in Perl (1987) you use the word `sub`. In C-family languages C (1972), C++ (1985), C# (2000), also in Java (1995) you don't need any word to declare it: return_type function_name(args) In languages from the last decade you have Rust (2010) with the keyword `fn`, Kotlin with `fun`. Dart (2011) follows C-family style and Julia (2012) offers two ways of defining a named function: using the word `function` or the compact version with no keyword `f(x,y) = x + y`. Following PEP20 <https://www.python.org/dev/peps/pep-0020/> aphorism of `Simple is better than complex.` I would say no word like C-family is simpler. But there is also `Explicit is better than implicit.`. I would choose `fun` or even `fn` since they at least resemble the word function. Eg: >>> def x(args): ... pass would turn into: >>> fun x(args): ... pass Resemblance of function leads me to my next point. 2. Class name Again, functions are an exception. >>> x = 1 >>> type(x) <class 'int'> *# not integer* >>> x = {"a": 1} >>> type(x) <class 'dict'> *# not dictionary* >>> def x(args): ... pass ... >>> type(x) <class 'function'> *# not fun =(* 3. Type hinting This is when the syntax gets trickier. With PEP 585 <https://www.python.org/dev/peps/pep-0585/> you don't have to do: >>> from typing import Dict, List, Set to have: >>> def x(foo: dict[string, int], bar: list[float]) -> None: ... pass But if you have a function as a parameter, your only option is to use Callable. It breaks the rule again having a totally different syntax: >>> from typing import Callable >>> def x(method: Callable[[int, dict], None]) -> None: ... pass So you have `def`, `function`, `Callable` and also this [[args], return] syntax, also different from the rest of type hinting syntax. I would add a `fun` type without the need of an import. Also Callable syntax could also be simplified with the -> operator into something like: >>> def x(method: fun[int, dict] -> None) -> None: ... pass === Preliminary Proposal === With the same goal of removing the pain and making life of Python teachers easier (PEP 585 <https://www.python.org/dev/peps/pep-0585/>), I would change this: >>> from typing import Callable >>> def x(method: Callable[[int, dict], None]) -> None: ... pass ... >>> type(x) <class 'function'> Into something like this: >>> fun x(method: fun[int, dict] -> None) -> None: ... pass ... >>> type(x) <class 'fun'> In order to make this possible the following changes would be needed: 1. adding `fun` keyword to method definition (maintaining `def` for backwards compatibility); 2. adding `fun` built-in type in addition to Callable; 3. supporting `->` as return syntax for type hinting. Isn't it much more `fun` to learn the language this way? Kind regards, Thiago Carvalho D'Ávila

4 months, 1 week

encode/decode for all objects

by Michael A. Smith

I was playing with the codecs module and realized that there's untapped potential to use them for de/serialization. It's easy enough to register a codec for some case, but very few objects (I think only strings and bytes and their stream cousins) have native encode/decode methods. It seems to me that obj.encode("json") and str.decode("json"), for example, would be a powerful feature, if it were tied into the native codecs registry, enabling users to simplify a lot of serialization code and implement or tie-in any codec that makes sense. Right now, if I want to json.dumps a MappingProxyType, I believe I have to pass a custom JSONEncoder to json.dumps explicitly every time I call it. But I think I should be able to register one, and then just call thing.encode('json'). I could call codecs.encode(thing, 'json'), but I think maybe I shouldn't have to import codecs or json into my modules to do this. What do you think? In case anyone is interested, here's a simple registration of json as a codec that works today: import codecs, json def encode(obj): try: size = len(obj) except TypeError: size = 1 return json.dumps(obj), size def decode(obj): return json.loads(obj), len(obj) codec_info = codecs.CodecInfo( name='json', encode=encode, decode=decode ) codecs.register({'json': codec_info}.get) print(codecs.encode({'a':1}, 'json')) # etc

4 months, 1 week

isolating user allocations

by wenjunhuang＠umass.edu

Hello, The API provided by PEP 445 makes it possible to intercept allocation requests through hooks, but it seems that both user allocations and interpreter allocations are sent to the hooks. Here, user allocations refer to those that are triggered explicitly by the code (e.g. memory allocations to hold the integer created by x = 1), and interpreter allocations refer to everything else (e.g. memory allocations for internal states). I've poked around a bit in the interpreter source code, and I think such differentiations aren't being done at all, so all allocations are directed to the same set of API. If that's indeed the case, why is the interpreter implemented this way? Would it make sense to implement the differentiation? Thanks

4 months, 1 week

Official joint communication from the Python Steering Council, PSF Board of Directors, and PSF Conduct WG

by Brett Cannon

[This has been sent to python-committers, python-dev, and python-ideas] Recently, a series of discussions on this mailing list resulted in behavior that did not live up to the standards of the Python Community. The PSF Board of Directors, Python Steering Council, and the PSF Conduct Working Group would like to remind this community that our shared goal is to advance the Python language while simultaneously building a diverse, inclusive, and welcoming Python community. While the community has made progress in recent years, this discussion made it clear to many of us that we still have room to grow. At the request of the PSF Board, the Steering Council, and members of the community, the PSF Conduct WG met to discuss these recent events and recommend action. As a result, warnings will be given to several members of the Python Community, and the Steering Council will be taking further action. Especially during passionate discussions like these, we'd like to ask that all Pythonistas focus on being welcoming and respectful, and that we all try to act in the best spirit of the Python Community. Thank you, Python Steering Council PSF Board of Directors PSF Conduct WG

4 months, 1 week

Pickle security improvements

by Random832

The current practice, by overriding find_class, is limited to overriding what globals get loaded. This makes it impossible to distinguish globals that will be used as data from globals that will be called as constructors, along with similar concerns with object attributes [especially methods] obtained by loading builtins.getattr as global. I would suggest also exposing for overrides the points where a callable loaded from the pickle is called - on the pure-python _Unpickler these are _instantiate, load_newobj, load_newobj_ex, and load_reduce, though it might be worthwhile to make a single method that can be overridden and use it at the points where each of these call a loaded object.

4 months, 1 week

Restricting Python

by Robert

Hi, I’ve been lurking for a while. This is my first real post in a long time. This is a proposal for a system-less python; that is, a version of python that does not have file or other inappropriate access to the os. The idea is to provide a python environment capable of safely running insecure scripts. The inspiration and obvious use case for this is an addition to or replacement for Javascript scripting in a web browser. Another use might be for scripting a spreadsheet, such as Excel. I do not think this needs to be a separate version of Python. The disable feature could be controlled by a command line flag, which would also have other uses described below. For an embedded system, this could be compiled in somehow. I don’t think the changes to the language would be all that great. The only built in change would be disabling the open function call, so there could be no local file access. The next issue is the standard library, a lot of which would have to be disabled. Certainly the os and subprocess modules would not be allowed. On the other side, there should be no restrictions on, for example, collections or strings. The biggest problem is third party imports. We can’t just disallow them. The whole point of a browser plug in would be to allow gui. What I have in mind is a means whereby imports could be white-listed (apologies for my un-pc phrasing.) That same command line switch (maybe -r for restricted) could take an argument for a special python syntax configuration file which would be loaded before other processing is done, and would be unalterable, and perhaps invisible, to the program scripts. from restrict import allow_imports, import_config, python_config allow_imports (( ‘collections’, ‘wx’, ‘pandas’, )) etc. Imports not specified in the list would be disallowed by the import mechanism. White-listing imports would also have uses for a safe pickle system. Note that this doesn’t break any existing code. No command line switch, business as usual, everything; command line switch, import restrictions and configurations apply. I think once we add such a configuration file, we’ll find all sorts of other uses for it. One thought is a generalized, and again read-only, means of configuring all imports: import_config (‘pandas’ : {‘allow_disk’ : False,}) import_config (‘wx’ : {‘allow_file_dialogs’ : False,}) import_config (‘subprocess’ : {‘allow_only’ : ‘/safe_directory’,}) If an import module had no configuration in the config file, nothing would would change. If it does, the import mechanism would somehow pass the config dictionary to the import start up. I’m not jived well enough on the mechanics of import to know how this would work, but again, it wouldn’t break existing code, as if there’s no config, nothing changes. The python system configuration could go in the same file: python_config ({‘allow_open’ : False}) The generalized case of restricting various systems has a whole lot of cases. For a web browser, you might want to restrict disk access. For a different application, you might want no gui or network access. Much of this could be flushed out later. We’d probably want to add two new exceptions: IllegalOpen, for an open call when restricted, and UnsupportedImport for an unauthorized import. I’m thinking for an interpreter embedded in a web browser, there would a special import like in embedded systems now: import firefox as ff ff.web_import (‘url’, ‘module’) ff.gdi.drawtext (‘mystring’) Input = ff.input.input_string () A standard import would import the package from a local system, including binary modules. A special web_import might import a python-only module from a web server. For the first step, and following KISS, I see the following changes. 1)Allow the open built in function to be conditionally disabled. 2)Add the command line switch and read only configuration for restricting allowed imports. 3)Sort through the standard library to figure out what should be allowed in a system-less configuration and what should not. 4)Add appropriate exceptions This concept would require a whole lot of changes down the road, particularly to binary imports; that would have to be flushed out after the initial changes, and by third party module developers. That’s it; thank you for your consideration. Robert Kaplan

4 months, 1 week

Use lambda in str.replace/re.sub

by artem6191

Example: '1 1 1 1'.replace('1', lambda char, index: str(index)+char) # '01 11 21 31'

4 months, 1 week

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas July 2020