Mailman 3 September 2019 - Python-ideas

Dictionary Keys treated as Sets
by Rupert Spann 17 Sep '19

17 Sep '19

related to: pep-0584, PEP-0218, and ??? The request is that since dictionary keys can not be duplicated / are unique that they may behave the same as sets - and have the set operators actions and return sets. eg.: aSet = set('a','b','c','bar') aDict = {'a':'aaa', 'b':'baa' , 'foo':fou' } bDict = {'a':'ea', 'b':'bee' , 'foo':fuh', 'bar': 'drink' } aSet | aDict | bDict = set('a','b','c','foo','bar') bDict & aDict =set('a','b','foo') bDict - aSet =set('foo') aDict^bDict =set('bar') The dictionary class would support set filtering: New_dict_from_set = { your_key: old_dict[your_key] for your_key in aSET } e.g. a function such as: (taking the intersection) aDict.Subdictionary(aSet) ={'a':'aaa', 'b':'baa'} set(dictionary) would return the Keys (as a mutable). Adding an element to the mutable set would add a None as a value to the dictionary. The "-=", "-", "+", "+=" would behave as specified in pep-0584 if both are dictionaries, but if one of the operands is a set - a set is returned. The behaviour of bDict.Subdictionary(aDict) = {'a':'ea', 'b':'bee' , 'foo':fuh') i.e. treats aDict as a set aDict.Subdictionary(List)== aDict.Subdictionary(Set(List))

5 4

Cache an entire import object graph
by Anders Hovmöller 16 Sep '19

16 Sep '19

Hi, I would like some feedback on the feasibility of an idea I've had: caching an entire import graph to speed up interactive apps. The idea is that tools like pytest, pip, hg, etc would be able to save and restore an import graph to cut down on initial startup time. As I think of it, this will be an opt-in system where a cache file could be stored on disk and loaded quickly and then gone through with one or two passes to relocate pointers to the present values if needed. The details of the API to be determined later. I'm thinking it would require changes to pip/setuptools to clear these caches when a package is upgraded for example. I have very limited experience with the CPython code base so I'd like to know if this is even a feasible thing to do? Best regards Anders

4 5

Feature Request: Forward-Pipe in Operator in Python
by Juan Telleria 15 Sep '19

15 Sep '19

Dear Python-Ideas Mailing List, The R Programming Language has for function chaining: https://magrittr.tidyverse.org/ And the most similar package we have in Python till now is: https://sspipe.github.io/ Anyone would be willing to work on this feature, making it more mature, stable and straighforward, so that it can be included natively in Python in the future? Thank you! Juan

1 0

Re: Support for atomic types in Python
by Vinay Sharma 14 Sep '19

14 Sep '19

Hi, I apologise for my poor explanation of what I mean. I should have been more comprehensive in describing the idea. First of all I am only concerned with synchronization across multiple processes only, and not threads. Therefore I never mentioned multi-threading in my original post. I am familiar with GIL and understand that atomicity doesn't make sense across threads. Also, I didn't mean simple variables to be atomic. I just mean that some objects which are frequently being used across processes in multiprocessing can have some atomic operations/methods. Let's take the following example: import time from multiprocessing import Process, Value, Lock def func(val, lock): for i in range(50): time.sleep(0.01) with lock: val.value += 1 if __name__ == '__main__': v = Value('i', 0) lock = Lock() procs = [Process(target=func, args=(v, lock)) for i in range(10)] for p in procs: p.start() for p in procs: p.join() print(v.value) Because as far as I can perceive none of the types or operations you mention have any race conditions in multi-threaded Python code, and for multi-process or async code that is not a problem by design but for data designed to be shared, like memory mapped stuff (and then locks are the only way to go anyway). In the above code snippet, I have to use a lock so as to prevent data races, and using lock is quite straightforward and simple. But in cases where I have to synchronize let's say 5 different values. I will have to use multiple locks for synchronization, as using a single lock will be very slow, and I will have to wait if a lock has been acquired to update any one of the 5 values. Also, as far as I know (might be wrong) Value is stored in shared memory and is therefore very fast also. So, what I am proposing is a similar object to value to which operations like add, sub, xor, etc are atomic. Therefore, I wouldn't have to use lock at all for synchronization. Updates to these values can be made very easily by using calls such as __sync_and_and_fetch(), even when they are stored in a shared memory segment, accessible across processes. On 9 September 2019 at 9:59 AM, "Joao S. O. Bueno" <jsbueno(a)python.org.br> wrote: May I ask how acquainted you ar with parallel code, including multi-threading, in Python? Because as far as I can perceive none of the types or operations you mention have any race conditions in multi-threaded Python code, and for multi-process or async code that is not a problem by design but for data designed to be shared, like memory mapped stuff (and then locks are the only way to go anyway). Primitive data types like numbers and strings are imutable to startwith, so it is "very impossible" for multi-threading code to be an issue when using then. As for composite native data types like dicts, lists and sets, that could theoretically be left in an inconsistent state, that is already taken care of by the GIL. And even for user designed types, it is a matter of adding a threading lock inside the proper dunder methods, and it also fixes the issue for the users of those types. The only time I had to worry about a lock in a data type in Python (i.e., not a program wide state that is inherent to my problem), was when asking a novice question on "how to rename a dict key" - and even them, it felt a bit overkill. In other words, I may be wrong, but I think you are talking about a non-issue here. On Sun, 8 Sep 2019 at 11:33, Vinay Sharma via Python-ideas <python-ideas(a)python.org> wrote: Currently, C++ has support for atomic types, on which operations like add, sub, xor, etc can be done atomically, thereby avoiding data races. Having such a support will be very helpful in Python. For instance users won't have to use Locks for synchronising shared variables in case of multiprocessing. And, directly passing these atomic objects would be enough. These objects can have methods such as add, sub, etc to change the underlying variable. I believe this would make python much more easier for users in both trivial and complicated cases, without having to use lock. gcc also provides inbuilt support for some atomic operations. Documentation for the same is available at https://gcc.gnu.org/onlinedocs/gcc-5.2.0/gcc/_005f_005fsync-Builtins.html#g… Moreover, gcc also has support for Memory Model aware atomic operations documented at https://gcc.gnu.org/onlinedocs/gcc-5.2.0/gcc/_005f_005fatomic-Builtins.html… _______________________________________________ Python-ideas mailing list -- python-ideas(a)python.org To unsubscribe send an email to python-ideas-leave(a)python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/S4EV6… Code of Conduct: http://python.org/psf/codeofconduct/ _______________________________________________ Python-ideas mailing list -- python-ideas(a)python.org To unsubscribe send an email to python-ideas-leave(a)python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/GYYUC… Code of Conduct: http://python.org/psf/codeofconduct/

10 27

Re: [Python-Dev] Feature Request: Python Pipes: Incluye sspipe module
by Andrew Svetlov 14 Sep '19

14 Sep '19

I prefer keeping it as a separate library on PyPI. On Sat, Sep 14, 2019 at 12:26 AM Juan Telleria <jtelleriar(a)gmail.com> wrote: > > Could sspipe module be included as part of Python's Standard Library? > > https://sspipe.github.io/ > > https://github.com/sspipe/sspipe > > https://pypi.org/project/sspipe/ > > sspipe allows to use syntax such as: > > from sspipe import p, px > import numpy as np > import pandas as pd > > ( > np.linspace(0, pi, 100) > | p({'x': px, 'y': np.sin(px)}) > | p(pd.DataFrame) > | px[px.x > px.y].head() > | p(print, "Example 6: pandas and numpy support:\n", px) > ) > > The issue in Python's Bug Tracker is: > > https://bugs.python.org/issue38052 > > Any Core Python Developer willing to support this PEP? > > Thank you, > > Juan Telleria > > _______________________________________________ > Python-Dev mailing list -- python-dev(a)python.org > To unsubscribe send an email to python-dev-leave(a)python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/2MUGDTK… -- Thanks, Andrew Svetlov

1 0

PEP-603: Adding a frozenmap type to collections
by Yury Selivanov 13 Sep '19

13 Sep '19

Hi! I've just published a PEP to add a frozenmap type to Python; it should be online shortly. Read it here: https://discuss.python.org/t/pep-603-adding-a-frozenmap-type-to-collections… Here's an excerpt from the Abstract: A *persistent data structure* is defined as a data structure that preserves the previous version of the data when the data is modified. Such data structures are effectively *immutable*, as operations on them do not update the structure in-place, but instead always yield a new updated structure (see [0]_ for more details.) This PEP proposes to add a new fully persistent and immutable mapping type called ``frozenmap`` to the ``collections`` module. The bulk of ``frozenmap``'s reference implementation is already used in CPython to implement the ``contextvars`` module. Yury

7 11

Conditional dict declarations
by rls＠hwyl.org 10 Sep '19

10 Sep '19

Hi list I have a proposal for a minor (improvement?) to dict declarations. I find a common requirement I have is to include one or more entries in a dict only if some condition is met. Currently, the usual way of doing that is to add one or more if statements. Simple (but hopefully relatable-to-real-world-code) example. Here we're constructing a query for a database based on the arguments passed in. In some cases we don't want to include some fields in the search at all depending on what the values passed in are. ### def construct_query(user_id, max_results=10000, active=None, deleted=None): query = { 'user_id': user_id } if max_results > 0: query['max_results'] = max_results if active is not None: query['active'] = active if deleted is not None: query['deleted'] = deleted return query ### This is OK-ish, but seems unnecessarily verbose, and it gets worse if you have more conditional entries. In addition, you're separating the dict setup code into two or more places so it is slower to grasp mentally. This is a fairly common (tedious) pattern in my code, but I feel it could be avoided. In my view it would be better if it was possible to conditionally add the entry in the dict declaration. My proposal is that there should be a special value defined that results in a dict entry not being added to the dict if either the key or the value equals this value. There may be several ways to achieve this, but here's one way it could work. ### def construct_query(user_id, max_results=10000, active=None, deleted=False): return { 'user_id': user_id, 'max_results': max_results or dict.skip, 'active': active if active is not None else dict.skip, 'deleted': deleted if deleted is not None else dict.skip } ### ..."skip" being a unique object defined in and accessible from dict (there may be better places to make it available). The dict insertion code then simply ignores those entries when creating the dict. I think this would be relatively easy to achieve. No significant modification to the language, just a minor modification to the dict code (as far as I know).

13 23

Automatic translation of Python to assembly language
by Mark ＠pysoniq 09 Sep '19

09 Sep '19

Hello to the list, I have an idea for Python that is non-traditional in that it doesn’t extend or modify existing Python language structure. The idea uses Python to translate Python, entirely under program control, directly to optimized assembly language .dll or .so files, called “extensions.” Extensions are called from Python using Python’s ctypes interface. The ctypes wrapper for each extension is created automatically. The goal of this idea is for Python to perform as fast or faster than C or C++, without leaving Python. Details: 1. Large project – now 23,556 lines of Python. 2. Evolved from a project that automatically translated APL to assembly language dlls -- more than 30,000 hours of development in APL and assembly language. 3. Solves the tremendous problem of coding assembly by hand. 4. Point-and-click interface. 5. Ahead-of-time compilation. 6. Python translated directly to assembly language – no third-party compiler (GCC, LLVM, Clang, etc.) or intermediate representation. 7. Advanced assembly language optimizations: registers, SIMD, multicore, loop fusion, loop unrolling, etc., custom-fitted to Python. 8. No Global Interpreter Lock issues – ctypes releases the GIL. Extensions have full use of all threads and cores. 9. NumPy and SciPy functions, as well as Python built-in functions and built-in library functions, translated directly to optimized assembly language to avoid expensive Python callbacks. 10. Memory safe: a. controls buffer access and frees every memory pointer when the extension returns from assembly to Python b. handles bounds checking on variables and arrays passed into the extension by ctypes c. extensions will not encounter errors such as buffer overflows, buffer over-reads, or memory race conditions d. handles recursive programs with its own stack, thus avoiding stack exhaustion for recursive programs More details at https://PysoniQ.com: A video demonstration Try out the point-and-click interface at the “Try PysoniQ” link A detailed Project Overview Technical FAQs Blog and speed metric links for deeper analysis of the technologies Downloadable PDFs – see the Resources link Any comments from the Python community on this project would be most appreciated! Thank you. Mark mark(a)pysoniq.com

10 39

Inspired by Scala, a new syntax for Union type
by Philippe Prados 09 Sep '19

09 Sep '19

Hello everybody, Scala 3 propose the a new syntax for Union type. See here <https://dotty.epfl.ch/docs/reference/new-types/union-types.html>. I propose to add a similar syntax in Python. # Operator for Union assert( int | str == Union[int,str]) assert( int | str | float == Union[int,str,float]) # Operator for Optional assert( ~int == Optional[int]) Now, it's possible to write: def fn(bag:List[int | str], option: ~int = None) -> float | str: ... in place of def fn(bag:List[Option[int,str]], option: Optional[int] = None) -> Union[float,str]: ... I think these syntaxes are more clear, and can help with the adoption of typing. I test and implement these ideas in a two fork : One for CPython <https://github.com/pprados/cpython> and one for MyPy <https://github.com/pprados/mypy>. See the branches add_OR_to_types (for Union syntax) or add_INVERT_to_types (for Union and Optional syntax). How I implement that ? I add the operators __or__ and __revert__ to PyType_Type. The C code is similar of : from typing import * def type_or(self,right): return Union[self,right] type(type).__or__ = type_or Actually, the accepted syntax for typing is : annotation: name_type name_type: NAME (args)? args: '[' paramslist ']' paramslist: annotation (',' annotation)* [','] I propose to extend the syntax to : annotation: ( name_type | or_type | invert_type ) name_type: NAME (args)? args: '[' paramslist ']' paramslist: annotation (',' annotation)* [','] or_type: name_type '|' annotation invert_type: '~' annotation What do you think about that ? The draft of a PEP is here <https://github.com/pprados/peps/blob/master/pep-9999.rst>. Regards

15 75

Support for atomic types in Python
by Vinay Sharma 09 Sep '19

09 Sep '19

Currently, C++ has support for atomic types, on which operations like add, sub, xor, etc can be done atomically, thereby avoiding data races. Having such a support will be very helpful in Python. For instance users won't have to use Locks for synchronising shared variables in case of multiprocessing. And, directly passing these atomic objects would be enough. These objects can have methods such as add, sub, etc to change the underlying variable. I believe this would make python much more easier for users in both trivial and complicated cases, without having to use lock. gcc also provides inbuilt support for some atomic operations. Documentation for the same is available at https://gcc.gnu.org/onlinedocs/gcc-5.2.0/gcc/_005f_005fsync-Builtins.html#g… Moreover, gcc also has support for Memory Model aware atomic operations documented at https://gcc.gnu.org/onlinedocs/gcc-5.2.0/gcc/_005f_005fatomic-Builtins.html…

2 1