Mailman 3 August 2019 - Python-ideas

Adding a thin wrapper class around the functions in stdlib.heapq
by bunslow Feb. 18, 2022

Feb. 18, 2022

Nothing so bombastic this time. The heapq functions are basically all named "heapsomething", and basically all take a "heap" for their first argument, with supplementary args coming after. It's a textbook example of the (hypothetical) Object Oriented Manifesto™ where defining a class increases type safety and programmers' conceptual clarity. There're practically no drawbacks, and the code to be added would be very simple. Updating the tests and docs would probably be harder. In pure Python, … [View More]

17 32

Make fnmatch.filter accept a tuple of patterns
by Andre Delfino March 26, 2021

March 26, 2021

Frequently, while globbing, one needs to work with multiple extensions. I’d like to propose for fnmatch.filter to handle a tuple of patterns (while preserving the single str argument functionality, alas str.endswith), as a first step for glob.i?glob to accept multiple patterns as well. Here is the implementation I came up with: https://github.com/python/cpython/compare/master...andresdelfino:fnmatch-mu… If this is deemed reasonable, I’ll write tests and documentation updates. Any opinion?

9 15

Dataclasses, keyword args, and inheritance
by George Leslie-Waksman March 12, 2021

March 12, 2021

The proposed implementation of dataclasses prevents defining fields with defaults before fields without defaults. This can create limitations on logical grouping of fields and on inheritance. Take, for example, the case: @dataclass class Foo: some_default: dict = field(default_factory=dict) @dataclass class Bar(Foo): other_field: int this results in the error: 5 @dataclass ----> 6 class Bar(Foo): 7 other_field: int 8 ~/.pyenv/versions/3.6.2/envs/… [View More]clover_pipeline/lib/python3.6/site-packages/dataclasses.py in dataclass(_cls, init, repr, eq, order, hash, frozen) 751 752 # We're called as @dataclass, with a class. --> 753 return wrap(_cls) 754 755 ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in wrap(cls) 743 744 def wrap(cls): --> 745 return _process_class(cls, repr, eq, order, hash, init, frozen) 746 747 # See if we're being called as @dataclass or @dataclass(). ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in _process_class(cls, repr, eq, order, hash, init, frozen) 675 # in __init__. Use "self" if possible. 676 '__dataclass_self__' if 'self' in fields --> 677 else 'self', 678 )) 679 if repr: ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in _init_fn(fields, frozen, has_post_init, self_name) 422 seen_default = True 423 elif seen_default: --> 424 raise TypeError(f'non-default argument {f.name!r} ' 425 'follows default argument') 426 TypeError: non-default argument 'other_field' follows default argument I understand that this is a limitation of positional arguments because the effective __init__ signature is: def __init__(self, some_default: dict = <something>, other_field: int): However, keyword only arguments allow an entirely reasonable solution to this problem: def __init__(self, *, some_default: dict = <something>, other_field: int): And have the added benefit of making the fields in the __init__ call entirely explicit. So, I propose the addition of a keyword_only flag to the @dataclass decorator that renders the __init__ method using keyword only arguments: @dataclass(keyword_only=True) class Bar(Foo): other_field: int --George Leslie-Waksman [View Less]

12 35

Add command-line option to unittest for enabling post-mortem debugging
by Dominik Vilsmeier Jan. 9, 2021

Jan. 9, 2021

Consider the following example: import unittest def foo(): for x in [1, 2, 'oops', 4]: print(x + 100) class TestFoo(unittest.TestCase): def test_foo(self): self.assertIs(foo(), None) if __name__ == '__main__': unittest.main() If we were calling `foo` directly we could enter post-mortem debugging via `python -m pdb test.py`. However since `foo` is wrapped in a test case, `unittest` eats the exception and thus prevents post-… [View More]

1 1

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley Dec. 16, 2020

Dec. 16, 2020

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists … [View More]now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich [View Less]

10 15

Add the imath module
by Serhiy Storchaka March 22, 2020

March 22, 2020

What are your thoughts about adding a new imath module for integer mathematics? It could contain the following functions: * factorial(n) Is just moved from the math module, but non-integer types are rejected. Currently math.factorial() accepts also integer floats like 3.0. It looks to me, the rationale was that at the time when math.factorial() was added, all function in the math module worked with floats. But now we can revise this decision. * gcd(n, m) Is just moved from the math … [View More]

14 38

New explicit methods to trim strings
by Alex Grigoryev March 21, 2020

March 21, 2020

Following the discussion here (https://link.getmailspring.com/link/7D84D131-65B6-4EF7-9C43-51957F9DFAA9@ge…) I propose to add 3 new string methods: str.trim, str.ltrim, str.rtrim Another option would be to change API for str.split method to work correctly with sequences. In [1]: def ltrim(s, seq): ...: return s[len(seq):] if s.startswith(seq) else s ...: In [2]: def rtrim(s, seq): ...: return s[:-len(seq)] if s.endswith(seq) else s ...: In [3]: def trim(s, seq): ...: return ltrim(rtrim(s, … [View More]

28 162

Re: [Python-ideas] Allow star unpacking within an slice expression
by Neil Girdhar March 7, 2020

March 7, 2020

I didn't think of this when we were discussing 448. I ran into this today, so I agree with you that it would be nice to have this. Best, Neil On Monday, December 4, 2017 at 1:02:09 AM UTC-5, Eric Wieser wrote: > > Hi, > > I've been thinking about the * unpacking operator while writing some numpy > code. PEP 448 allows the following: > > values = 1, *some_tuple, 2 > object[(1, *some_tuple, 2)] > > It seems reasonable to me that it should be extended to … [View More]

2 2

Specify number of items to allocate for array.array() constructor
by Sven Rahmann Feb. 22, 2020

Feb. 22, 2020

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int … [View More]

14 20

Fixed point format for numbers with locale based separators
by Łukasz Stelmach Dec. 20, 2019

Dec. 20, 2019

Hi, I would like to present two pull requests[1][2] implementing fixed point presentation of numbers and ask for comments. The first is mine. I learnt about the second after publishing mine. The only format using decimal separator from locale data for float/complex/decimal numbers at the moment is "n" which behaves like "g". The drawback of these formats, I would like to overcome, is the inability to print numbers ranging more than one order of magnitude with the same number of decimal digits … [View More]

5 9

PEP's shouldn't require a sponsor
by Batuhan Taskaya Dec. 3, 2019

Dec. 3, 2019

Why do i need to convince a core developer for my PEP? AFAIK the steering council can include non core developers (i know it isn't that current case but for the future this is important). And if the last authority who will approve my PEP is the steering council i just need to convince them not core developers. Sponsors can stay (and they should because guidance is important) but thy shouldn't be mandatory. Let everyone to send their peps.

22 29

Please consider adding of functions file system operations to pathlib
by George Fischhof Nov. 16, 2019

Nov. 16, 2019

Good day all, as a continuation of thread "OS related file operations (copy, move, delete, rename...) should be placed into one module" https://mail.python.org/pipermail/python-ideas/2017-January/044217.html please consider making pathlib to a central file system module with putting file operations (copy, move, delete, rmtree etc) into pathlib. BR, George

3 3

Start argument for itertools.accumulate() [Was: Proposal: A Reduce-Map Comprehension and a "last" builtin]
by Raymond Hettinger Oct. 23, 2019

Oct. 23, 2019

> On Friday, April 6, 2018 at 8:14:30 AM UTC-7, Guido van Rossum wrote: > On Fri, Apr 6, 2018 at 7:47 AM, Peter O'Connor <peter.ed...(a)gmail.com> wrote: >> So some more humble proposals would be: >> >> 1) An initializer to itertools.accumulate >> functools.reduce already has an initializer, I can't see any controversy to adding an initializer to itertools.accumulate > > See if that's accepted in the bug tracker. It did come-up once but was closed … [View More]for a number reasons including lack of use cases. However, Peter's signal processing example does sound interesting, so we could re-open the discussion. For those who want to think through the pluses and minuses, I've put together a Q&A as food for thought (see below). Everybody's design instincts are different -- I'm curious what you all think think about the proposal. Raymond --------------------------------------------- Q. Can it be done? A. Yes, it wouldn't be hard. _sentinel = object() def accumulate(iterable, func=operator.add, start=_sentinel): it = iter(iterable) if start is _sentinel: try: total = next(it) except StopIteration: return else: total = start yield total for element in it: total = func(total, element) yield total Q. Do other languages do it? A. Numpy, no. R, no. APL, no. Mathematica, no. Haskell, yes. * http://docs.scipy.org/doc/numpy/reference/generated/numpy.ufunc.accumulate.… * https://stat.ethz.ch/R-manual/R-devel/library/base/html/cumsum.html * http://microapl.com/apl/apl_concepts_chapter5.html \+ 1 2 3 4 5 1 3 6 10 15 * https://reference.wolfram.com/language/ref/Accumulate.html * https://www.haskell.org/hoogle/?hoogle=mapAccumL Q. How much work for a person to do it currently? A. Almost zero effort to write a simple helper function: myaccum = lambda it, func, start: accumulate(chain([start], it), func) Q. How common is the need? A. Rare. Q. Which would be better, a simple for-loop or a customized itertool? A. The itertool is shorter but more opaque (especially with respect to the argument order for the function call): result = [start] for x in iterable: y = func(result[-1], x) result.append(y) versus: result = list(accumulate(iterable, func, start=start)) Q. How readable is the proposed code? A. Look at the following code and ask yourself what it does: accumulate(range(4, 6), operator.mul, start=6) Now test your understanding: How many values are emitted? What is the first value emitted? Are the two sixes related? What is this code trying to accomplish? Q. Are there potential surprises or oddities? A. Is it readily apparent which of assertions will succeed? a1 = sum(range(10)) a2 = sum(range(10), 0) assert a1 == a2 a3 = functools.reduce(operator.add, range(10)) a4 = functools.reduce(operator.add, range(10), 0) assert a3 == a4 a4 = list(accumulate(range(10), operator.add)) a5 = list(accumulate(range(10), operator.add, start=0)) assert a5 == a6 Q. What did the Python 3.0 Whatsnew document have to say about reduce()? A. "Removed reduce(). Use functools.reduce() if you really need it; however, 99 percent of the time an explicit for loop is more readable." Q. What would this look like in real code? A. We have almost no real-world examples, but here is one from a StackExchange post: def wsieve(): # wheel-sieve, by Will Ness. ideone.com/mqO25A->0hIE89 wh11 = [ 2,4,2,4,6,2,6,4,2,4,6,6, 2,6,4,2,6,4,6,8,4,2,4,2, 4,8,6,4,6,2,4,6,2,6,6,4, 2,4,6,2,6,4,2,4,2,10,2,10] cs = accumulate(cycle(wh11), start=11) yield( next( cs)) # cf. ideone.com/WFv4f ps = wsieve() # codereview.stackexchange.com/q/92365/9064 p = next(ps) # 11 psq = p*p # 121 D = dict( zip( accumulate(wh11, start=0), count(0))) # start from sieve = {} for c in cs: if c in sieve: wheel = sieve.pop(c) for m in wheel: if not m in sieve: break sieve[m] = wheel # sieve[143] = wheel@187 elif c < psq: yield c else: # (c==psq) # map (p*) (roll wh from p) = roll (wh*p) from (p*p) x = [p*d for d in wh11] i = D[ (p-11) % 210] wheel = accumulate(cycle(x[i:] + x[:i]), start=psq) p = next(ps) ; psq = p*p next(wheel) ; m = next(wheel) sieve[m] = wheel [View Less]

17 49

Why not accept lists or arbitrary iterables in str.endswith?
by Soni L. Oct. 14, 2019

Oct. 14, 2019

I'm parsing configs for domain filtering rules, and they come as a list. However, str.endswith requires a tuple. So I need to use str.endswith(tuple(list)). I don't know the reasoning for this, but why not just accept a list as well?

3 6

Skip modules by default in star-import
by Serhiy Storchaka Sept. 19, 2019

Sept. 19, 2019

From the description of the import statement in the language reference [1]: > The *public names* defined by a module are determined by checking the module's > namespace for a variable named ``__all__``; if defined, it must be a sequence > of strings which are names defined or imported by that module. The names > given in ``__all__`` are all considered public and are required to exist. If > ``__all__`` is not defined, the set of public names includes all names found > in the … [View More]module's namespace which do not begin with an underscore character > (``'_'``). ``__all__`` should contain the entire public API. It is intended > to avoid accidentally exporting items that are not part of the API (such as > library modules which were imported and used within the module). As an example of names which should be excluded from the public API the reference mentions names of modules which were imported and used within the module. And they are perhaps the most common cases. It is often imported module names are the only non-underscored names which are imported by a star-import by accident. For example, modules re and sys were used in Lib/idlelib/editor.py but there were no imports for them. This worked because they were imported implicitly by `from tkinter import *`. [1] There is other case when a module name can be occurred in the module namespace: if it is a submodule. For example: >>> from xml.etree import * >>> ElementTree Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'ElementTree' is not defined >>> ElementPath Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'ElementPath' is not defined >>> import xml.etree.ElementTree >>> from xml.etree import * >>> ElementTree <module 'xml.etree.ElementTree' from '/home/serhiy/py/cpython3.8/Lib/xml/etree/ElementTree.py'> >>> ElementPath <module 'xml.etree.ElementPath' from '/home/serhiy/py/cpython3.8/Lib/xml/etree/ElementPath.py'> Names imported by a `from xml.etree import *` are depended on importing a xml.etree's submodule ElementTree. I do not think this is good. I propose to change the rule for determining the set of public names if `__all__` is not defined. In addition to underscored names I propose to exclude names of modules. In these rare cases where modules should be imported in a star import it is not difficult to introduce `__all__`. [1] https://docs.python.org/3/reference/simple_stmts.html#the-import-statement [2] https://bugs.python.org/issue29446 [View Less]

9 16

Inspired by Scala, a new syntax for Union type
by Philippe Prados Sept. 9, 2019

Sept. 9, 2019

Hello everybody, Scala 3 propose the a new syntax for Union type. See here <https://dotty.epfl.ch/docs/reference/new-types/union-types.html>. I propose to add a similar syntax in Python. # Operator for Union assert( int | str == Union[int,str]) assert( int | str | float == Union[int,str,float]) # Operator for Optional assert( ~int == Optional[int]) Now, it's possible to write: def fn(bag:List[int | str], option: ~int = None) -> float | str: ... in place of def fn(bag:List[Option[… [View More]

15 75

Add a `dig` method to dictionaries supporting the retrieval of nested keys
by j_livermont＠yahoo.com Sept. 4, 2019

Sept. 4, 2019

I did some searching to see if this has already been proposed before, but didn't find any evidence that it has. If so, let me know and I'll go away :) One of the tasks that I encounter frequently enough is retrieving a nested key from a nested collection of dictionaries (a dictionary of dictionaries that can be any number of layers deep). There are multiple ways of tackling this, normally done with `reduce()`, iterative looping or chained `get()`s and then handling `None`, `KeyError` or `… [View More]

9 9

Custom string prefixes
by stpasha＠gmail.com Sept. 3, 2019

Sept. 3, 2019

In Python strings are allowed to have a number of special prefixes: b'', r'', u'', f'' + their combinations. The proposal is to allow arbitrary (or letter-only) user-defined prefixes as well. Essentially, a string prefix would serve as a decorator for a string, allowing the user to impose a special semantics of their choosing. There are quite a few situations where this can be used: - Fraction literals: `frac'123/4567'` - Decimals: `dec'5.34'` - Date/time constants: `t'2019-08-26'` -… [View More]

20 94

Adding support for adequately tagging AIX (pep425) to support distributed wheels
by Michael Sept. 1, 2019

Sept. 1, 2019

Goal - using wheels rather than RPM and/or installp formats for distributing binary modules Why bother? One reason (there are likely more) - using wheels means packages/modules can be loaded in a virtualenv rather than require they are first loaded in the system environment using installp/rpm/yum (with root authority). This alone has been reason enough for me to do the research. Introduction The use of pip and wheels is commonplace in the worlds of Linux, macOS and Windows. Not so for AIX. … [View More]

3 2

adding support for a "raw output" in JSON serializer
by Richard Musil Aug. 29, 2019

Aug. 29, 2019

I have found myself in an awkward situation with current (Python 3.7) JSON module. Basically it boils down to how it handles floats. I had been hit on this particular case: In [31]: float(0.6441726684570313) Out[31]: 0.6441726684570312 but I guess it really does not matter. What matters is that I did not find a way how to fix it with the standard `json` module. I have the JSON file generated by another program (C++ code, which uses nlohmann/json library), which serializes one of the floats … [View More]

21 142

Add "Slice" type to typing module
by Dominik Vilsmeier Aug. 28, 2019

Aug. 28, 2019

Usually slices are created with integers such as `foo[1:5]`. However slices are not restricted to having integers for the `start, stop, step` parameters and thus can be used with any types. The most prominent example is probably `pandas` which allows slicing by index (and the index can be `str` for example): df = pd.DataFrame([[0], [1], [2]], index=list('abc'), columns=['x']) print(df.loc['a':'b']) # The first two items, 'a' and 'b' inclusive. In one of my projects I also employed … [View More]

2 1

Add boolean attribute to datetime objects stating if they are tz-aware
by Nick Timkovich Aug. 27, 2019

Aug. 27, 2019

Currently the datetime docs say: > An object of type time or datetime may be naive or aware. A datetime object d is aware if d.tzinfo is not None and d.tzinfo.utcoffset(d) does not return None. If d.tzinfo is None, or if d.tzinfo is not None but d.tzinfo.utcoffset(d) returns None, d is naive. A time object t is aware if t.tzinfo is not None and t.tzinfo.utcoffset(None) does not return None. Otherwise, t is naive. > https://docs.python.org/3/library/datetime.html#available-types Would … [View More]

2 1

[new idea] dynamic return with using function argument
by HUANG YUWEI Aug. 27, 2019

Aug. 27, 2019

Dear Python community, I am a heavy python user in numerical simulation. In python 3.8, we will have a new syntax `:=` that could assign values to variables as part of larger expression like, ``` if (n:=len(a)) > 10: .... ``` On the other hand, I also think that it would be useful if `:=` could be used in function argument, so that a local variable created inside a function could be returned **optionally**, for example, ``` def func(arg1, arg2, karg1=karg1_default, karg_return:=… [View More]

5 11

Idea/PEP draft: Add support of __main__.py to argparse.ArgumentParser.prog
by Michael Hooreman Aug. 25, 2019

Aug. 25, 2019

Hello, I have a proposal to improve support of __main__.py (python -m foobar) invocation to argparse.ArgumentParser You can find attached a PEP draft. Unfortunately, I'm not very confident on how to add that PEP and ... honnestly not very used to github. In short, can you help me and advise on this? Thanks a lot and best regards, Michaël Hooreman

4 23

PEP: Idiomatic API for shell scripting
by Thiago Padilha Aug. 21, 2019

Aug. 21, 2019

Python is often used a shell scripting language due to it being more robust than traditional Unix shells for managing complex programs, but it is significantly more verbose than Bash & friends for doing common shell scripting tasks such as writing pipelines and redirection . I propose adding an idiomatic shell scripting API that uses Python operator overloading in a way that allows invoking commands in a Bash-like way. A fully working implementation (which I've been using in my personal … [View More]scripts) can be found at https://github.com/tarruda/python-ush/. Here I will describe the basics of the API, more details in the github README.rst (which is also a doctest for the project). Here's how one can create commands: >>> from shell import Shell >>> sh = Shell() >>> cat = sh('cat') >>> ls = sh('ls') >>> sort = sh('sort') # or something like this: >>> cat, ls, sort = sh('cat', 'ls', 'sort') The Shell class is the entry point of the API and it is used as a factory for Command objects. For convenience we can add a builtin Shell instance which is a also module from where commands can be quickly imported from (idea taken from sh.py http://amoffat.github.io/sh/): >>> from shell.sh import cat, ls, sort We can construct more complex commands by calling it with certain arguments: >>> ls('-l', '-a', env={'LANG': 'C.UTF-8'}) And we can invoke the command by calling it without arguments: >>> ls() # or >>> ls('-l', '-a', env={'LANG': 'C.UTF-8'})() Invoking the command returns a tuple with the status code: >>> ls() (0,) The are more ways to invoke a command other than calling it without arguments. One such example is by iterating through it, which automatically takes care of piping the output back to python: >>> files = [] >>> for line in ls: ... files.append(line) Pipelines can be easily created with the `|` operator: >>> ls = ls('--hide=*.txt', env={'LANG': 'C.UTF-8'}) >>> sort = sort('--reverse') >>> ls | sort ls --hide=*.txt (env={'LANG': 'C.UTF-8'}) | sort --reverse Everything that can be done with Command objects, can also be done with Pipeline objects: >>> (ls | sort)() (0, 0) # single commands return a tuple to make the return value compatible with Pipelines >>> list(ls | sort) [u'tests', u'setup.cfg', u'pytest.ini', u'bin', u'README.rst'] We also use the Pipeline syntax for redirecting input/output. # piping to a filename string redirects output to the file >>> (ls | sort | '.stdout')() (0, 0) >>> str(cat('.stdout')) 'tests\nsetup.cfg\npytest.ini\nbin\nREADME.rst\n' # piping from a filename string redirects input from the file >>> str('setup.cfg' | cat) '[metadata]\ndescription-file = README.rst\n\n[bdist_wheel]\nuniversal=1\n' This is the basic idea. The current implementation only supports the classic subprocess API, but I can probably make it compatible with asyncio (to create async shells which spawns commands compatible with `await`). Thoughts? Is it worth writing a PEP for this? Best regards, Thiago. [View Less]

6 6

Re: [Distutils] Linux binary wheels?
by Wes Turner Aug. 20, 2019

Aug. 20, 2019

FWIW, conda supports the e.g. armv7l aarch32 and armv8 aarch64 / "ARM64" platforms. Third-party-built packages are the norm there; where there are channels like conda-forge and rpi. What does it mean to sign a CI build from a given unsigned git tag? "Build conda packages for ARM" https://github.com/conda-forge/conda-forge.github.io/issues/269 Raspbian is built for armv7. Raspberry Pi 2, 3, and 4 are armv8 CPUs (with hw SIMD, AES, SHA-1, SHA-256), so they support armv7l (aarch32) and armv8 (… [View More]

1 0

Re: [Distutils] Re: Linux binary wheels?
by Nick Timkovich Aug. 20, 2019

Aug. 20, 2019

On Tue, Aug 20, 2019, at 5:05 AM Matthew Brett <matthew.brett(a)gmail.com> wrote: > ... Unless you meant wheels for non-Intel platforms, in which case, > please do say more about you need. Minor tangent: I've seen some people use https://www.piwheels.org/ for Raspberry Pi (ARM 6/7), but could the ARM binaries be uploaded to PyPI? I think I'm conflating the wheel building spec (is manylinux amd64 specific, or as long as the libraries are on any architecture?), toolchains, … [View More]

1 0

Re: [Distutils] Re: Linux binary wheels?
by Brian Skinn Aug. 20, 2019

Aug. 20, 2019

On Tue, Aug 20, 2019 at 6:14 AM Matthew Brett <matthew.brett(a)gmail.com> wrote: > Hi, > > On Tue, Aug 20, 2019 at 11:10 AM Brian Skinn <brian.skinn(a)gmail.com> > wrote: > > > > > > > > On Tue, Aug 20, 2019, 06:05 Matthew Brett <matthew.brett(a)gmail.com> > wrote: > >> > >> Hi, > > > > > >> See the links that Wes posted for more details, or try pip installing > >> Numpy and Scipy in a new … [View More]

2 1

Re: [Distutils] Linux binary wheels?
by Wes Turner Aug. 20, 2019

Aug. 20, 2019

How does this proposal differ from manylinux2010? https://github.com/pypa/manylinux/blob/master/README.rst#example PEP 513: manylinux1 https://www.python.org/dev/peps/pep-0513/ PEP 571: The manylinux2010 Platform Tag (latest, as of 2019) https://www.python.org/dev/peps/pep-0571/ On Monday, August 19, 2019, Dan Stromberg <drsalists(a)gmail.com> wrote: > Hi folks. > I have a pair of ideas about Linux binary wheels, which are currently (I > heard) unsupported. > It seems … [View More]

1 0

Fwd: Optional kwarg for sequence methods in random module
by Patryk Gałczyński Aug. 19, 2019

Aug. 19, 2019

Hi! Recently I encountered a situation when I needed to pick a random population sample but with specific distribution function - paretovariate in this particular case. After poking around in the random module I found out that none of the so-called "sequence methods" methods support custom random function except for random.shuffle <https://github.com/python/cpython/blob/master/Lib/random.py#L293>. I would like to propose a new optional keyword argument for all these sequence methods (… [View More]

6 7

Re: Namespace context managers
by Niklas Rosenstein Aug. 18, 2019

Aug. 18, 2019

You may be interested in this working example of a @namespace decorator (for functions though, not for classes). https://stackoverflow.com/a/52358426/791713 — @namespace def mynamespace(): eggs = 'spam' class Bar: def hello(self): print("Hello, World!") assert mynamespace.eggs == 'spam' mynamespace.Bar().hello() — It uses sys.settrace() to find the frame of the function before it is called to convert it into an actual module object. My (largely underdeveloped and not recommended to be … [View More]

2 1

Extend ast with types for <type>* instead of using raw lists
by Caleb Donovick Aug. 17, 2019

Aug. 17, 2019

When walking an ast it impossible to know the type of an empty list without writing down some giant lookup from node types and field names to field types. More concretely it would nice be to able to programatically visit all blocks (stmt*) without having to something like: ``` class BlockVisitor(NodeVisitor): def visit_If(self, node: If): self.visit(node.test) self.visit_block(node.body) self.visit_block(node.orelse) def visit_FunctionDef(self, node: … [View More]

4 6

Add a `get_profile_dict` function to cProfile
by olshansky.daniel＠gmail.com Aug. 17, 2019

Aug. 17, 2019

Hey everyone, I've been using cProfile to profile some of my code, but found out that while the tools to print the results are very convenient, it was difficult to extract and send to a logging tool (i.e. fluentd) where it can be further parsed and analyzed. I'm thinking of adding a function to cProfile (https://github.com/python/cpython/blob/master/Lib/cProfile.py) similar to "get_profile_dict" below, but wanted to get some feedback from the community before I submit a proper PR for it. The … [View More]

1 0

Exception handling in objects
by Marat Sharafutdinov Aug. 15, 2019

Aug. 15, 2019

While using any library there is a need to handle exceptions it raises. In ideal conditions author of library defines base exception and builds hierarchy of exceptions based on it. Moreover, all exceptions a library raises (including those that do not inherit its base exception) are documented and this information is kept up to date. Unfortunately, this is not always the case. Library authors are free to use the language exception mechanism in any way. So instead of using documented features of … [View More]

6 6

hybrid implementation for PyLongObject (performance)
by malincns＠163.com Aug. 14, 2019

Aug. 14, 2019

The idea is mixing `PyLongObject` with `Python 2's PyIntObject` implementation. For example, on a 64-bit platform, if (an integer >=-9223372036854775808 and <=9223372036854775807), PyLongObject uses a native C type `signed long` to represent it. People mostly use +-* operations, maybe using native int is faster, even including the cost of overflow check. If operation will overflow or other operations like **, we can transform native int to current form, and run in current code path. Regards

4 4

Stdlib module - Draft
by Abdur-Rahmaan Janhangeer Aug. 13, 2019

Aug. 13, 2019

Greetings, Here is a draft after the stdlib module discussion <https://mail.python.org/archives/list/python-ideas@python.org/thread/XSYEVR…> . Abstract ======= Proposal: Adding a stdlib module to the standard library Defining the stdlib ============== Would the stdlib module be OS-independent or not? A better approach would be to go along the standard modules as defined in the docs <https://docs.python.org/3/library/index.html>. You get to know all modules within the stdlib, it … [View More]even lists curses on windows (see benefit at 2. below) with info. But it does not let you access it if you want to use it when importing. Same for the third issue: The third issue is alternative Python implementations, discussed at concerns no.2. Advantages of having a stdlib module ============================= Below are some benefits of having such a module: - 1 Knowing what module can be imported without installing This can be useful for interactive learning. Like with the inbuilt builtins library, one get to know what builtins are there in Python, though it's not the lib's only use. In the spirit of the keyword module, it can also check if an import forms part of the std lib - 2 Check for accidental name collisions It might come in handy to check for python platform collisions. An intersection between their modules will do the job - 3 Namespaced imports import math could be used as stdlib.math to better distinguish. Concerns ======== - 1. Backward compatibility Python 4.0 codes might contain codes incompatible with previous versions - 2. Different standard modules If a Python implementation adds in a module, would codes implementing stdlib.fancy_lib on a diffrent platform not get broken? That is essentially the same as executing curses on windows or RPi on Raspberry Pi - Raspian. Just like RPi codes would not run on a windows, similarly, stdlib.fancy_lib would not run on the standard C implementation. Ideas and Implementations ==================== - Andrew Barnert implemented a version of stdlib <https://github.com/abarnert/stdlib> where you can import without having to imply where the module is meaning instead of from x.y import z, you do: from stdlib import z - Each implementation could hardcode the names of std modules which could be accessed by stdlib for listing and imports - Use what help('modules') uses i.e pkgutil.walk_packages minus 3rd party modules - This stdlib implemenation <https://github.com/jackmaney/python-stdlib-list> from Jack Maney essentially hardcodes the libs like here <https://github.com/jackmaney/python-stdlib-list/blob/master/stdlib_list/lis…> . - The real implementation is up to ... be implemented. Draft containing inputs from: Andrew Barnert Chris Angelico Christopher Barker Steve Barnes Steven D'Aprano -- Abdur-Rahmaan Janhangeer pythonmembers.club <http://www.pythonmembers.club/> | github.com <https://github.com/Abdur-rahmaanJ> Mauritius [View Less]

2 2

Re: Idiomatic API for shell scripting
by Greg Price Aug. 13, 2019

Aug. 13, 2019

(Changed subject, as I don't think a PEP is the right focus for thinking about this problem space) Thiago Padilha wrote: > Python is often used a shell scripting language due to it being more > robust than traditional Unix shells for managing complex programs, but > it is significantly more verbose than Bash & friends for doing common > shell scripting tasks such as writing pipelines and redirection . I 100% agree with this problem statement. As others have mentioned, there … [View More]are several libraries people have written that try to do something in this direction. I’ve never found one that I felt approached the problem from quite the right angle, so I’ve been experimenting recently with my own solution (incorporating ideas from friends.) I guess this is a good prompt to discuss it more widely! All the ideas below are in the working implementation at https://github.com/gnprice/pysh . ---------- First, one unsung strength of Bash (and friends) that I think is actually an essential foundation is what happens when there’s no redirection, no pipelines, no fancy substitutions — you want to just run a program, and pass it some arguments. The core thing you need to do here is make a list of strings, for the command and its arguments. That’s how the underlying API works. So for example in Bash: # 5 strings: gpg -d -o "$cleartext_path" "$cryptotext_path" # (4 + len(commit_ids)) strings: git merge "${commit_ids[@]}" -m "$message" In Python this typically ends up looking like: subprocess.check_call(['gpg', '-d', '-o', cleartext_path, cryptotext_path]) subprocess.check_call(['git', 'merge'] + commit_ids + ['-m', message]) For the arguments that are literals — which often means most of them -- we keep saying quote-comma-space-quote, quote-comma-space-quote, effectively as a delimiter. That’s a lot of visual noise, as well as extra typing. And when you want to splice in a whole list of arguments, it gets worse. What we want here is something almost like this: subprocess.check_call( f'gpg -d -o {cleartext_path} {cryptotext_path}'.split()) # BAD! ... except that version steps into the classic shell pitfall where if a value happens to contain whitespace, it turns into several arguments and totally changes the meaning. But it’s so close. Really we just want to do that but with the `split` first — so it operates on the literal string you see in the source code — and *then* the `format`, or the f-string substitution. In fact you can implement a 70% solution in just a line: def shwords(fmt, **kwargs): return [word.format(**kwargs) for word in fmt.split()] >>> shwords('rm -rf {tmpdir}/{userdoc}', tmpdir='/tmp', userdoc='1 .. 2') ['rm', '-rf', '/tmp/1 .. 2'] A bit more work gives you f-string-like behavior, if you opt into it: subprocess.check_call(shwords_f('rm -rf {tmpdir}/{userdoc}')) check_cmd_f('rm -rf {tmpdir}/{userdoc}') # small convenience helper and some more gives you positional arguments: check_cmd('gpg -d -o {} {}', cleartext_path, cryptotext_path) and a format-minilanguage extension `{...!@}` , to substitute in a whole list: check_cmd('git merge {!@} -m {}', commit_ids, message) Full implementation here: https://github.com/gnprice/pysh/blob/8faa55b06/pysh/words.py As far as I know, this idea is a new one. I’ve found it goes a long way in making scripts that run a lot of external commands feel Pythonically concise and low-boilerplate. ---------- Then there are a number of features on top of that foundation that we need to really make interacting with external Unix-style commands as convenient as it can be. To avoid making this email longer I’ll just briefly gesture at two of them for now — more details in the repo! * Getting a command’s output in convenient form — not only on stdout, but success/failure from the return code: if not try_cmd('git diff-index --quiet HEAD'): raise ClickException("worktree not clean") # ... commit_id = try_slurp_cmd_f('git rev-parse --verify --quiet {name}') if commit_id is None: commit_id = try_slurp_cmd_f('git rev-parse --verify --quiet origin/{name}') if commit_id is None: raise MissingBranchException(name) * Pipelines. :-) The `|` operator really is too good to pass up: def first_mention(pattern): return pysh.slurp( cmd.run('git log --oneline --reverse --abbrev=9') | cmd.run('grep -m1 {}', pattern) | cmd.run('perl -lane {}', 'print $F[0]') ) I think one important direction for pipelines is making it convenient to stick bits of Python code in the middle of the pipeline, in amongst external programs. That direction isn’t fully developed in the current API. ---------- For anyone who finds this interesting, I’d encourage you to check out the demo scripts: https://github.com/gnprice/pysh/tree/master/example Then if you take some interesting fragment of script you have handy and try converting it to use this library, I’d be very curious to hear how it turns out! That’s what the demo/example scripts are for, and I think it’s the main ingredient this work needs at this stage: seeing how the API works out in a range of use cases, and seeing what patterns it can be improved to better serve. Also naturally I will be glad to discuss it here for as long as others are interested. Cheers, Greg [View Less]

1 0

Disallow all ending statements (continue/break/return) inside of finally
by Batuhan Taskaya Aug. 13, 2019

Aug. 13, 2019

About 2 days ago i was writing a blog post and i thought why dont i use a

1 1

Exceptions with Message Templates
by Ryan Fox Aug. 12, 2019

Aug. 12, 2019

Exception definitions in Python are not great. There are two main ways they're used in the code that I've read: 1) By far the most common: >>> class MyException(Exception): ... pass >>> raise MyException(f'Bad thing happened during {action} in {context}') 2) Much rarer: >>> class MyException(Exception): ... def __init__(self, action, context): ... super().__init__(f'Bad thing happened during {action} in {context}') >>> raise MyException(… [View More]

5 12

Namespace context managers
by Batuhan Taskaya Aug. 10, 2019

Aug. 10, 2019

I am proposing namespace context managers with implementing `__enter__` and `__exit__` on dict objects. It would make closures possible in python with a pythonic syntax. a = 4 namespace = {} with namespace: a = 3 assert a == 4 assert namespace["a"] == 3

15 32

Proposal: Use "for x = value" to assign in scope
by Peter O'Connor Aug. 9, 2019

Aug. 9, 2019

Alright hear me out here: I've often found that it would be useful for the following type of expression to be condensed to a one-liner: def running_average(x_seq): averages = [] avg = 0 for t, x in enumerate(x_seq): avg = avg*t/(t+1) + x/(t+1) averages.append(avg) return averages Because really, there's only one line doing the heavy lifting here, the rest is kind of boilerplate. Then I learned about the beautiful and terrible "for x in [value]": def … [View More]

5 4

Universal parsing library in the stdlib to alleviate security issues
by Nam Nguyen Aug. 9, 2019

Aug. 9, 2019

Hello list, I sent an email to this list two or three months ago about the same idea. In that discussion, there were both skepticism and support. Since I had some time during the previous long weekend, I have made my idea more concrete and I thought I would try with the list again, after having run it through some of you privately. GOAL: To have some parsing primitives in the stdlib so that other modules in the stdlib itself can make use of. This would alleviate various security issues we … [View More]

12 31

Re: adding support for a "raw output" in JSON serializer
by Peter Otten Aug. 8, 2019

Aug. 8, 2019

Richard Musil wrote: > I have found myself in an awkward situation with current (Python 3.7) JSON > module. Basically it boils down to how it handles floats. I had been hit > on this particular case: > > In [31]: float(0.6441726684570313) > Out[31]: 0.6441726684570312 > > but I guess it really does not matter. > > What matters is that I did not find a way how to fix it with the standard > `json` module. I have the JSON file generated by another program (C++ … [View More]

1 0

For-expression/throwaway comprehension
by Eli Berkowitz Aug. 7, 2019

Aug. 7, 2019

(This is my first time posting on any Python list; I've tried to search for this idea and didn't find it but if I looked in the wrong places/this has already been discussed I apologize and feel free to tell me!) Say you have a list and you want to perform some operation on each item in the list - but you don't need to store the result in a list. There are three simple ways of doing this, at least as far as I know: ([print(item)] could be any expression, just using it as an example) ``` lst = … [View More]

23 76

Making io.BufferedReader buffer available
by Jérôme Carretero Aug. 6, 2019

Aug. 6, 2019

Hi, I pushed http.client to a scenario where I would read streaming HTTP data (yeah, without going into native async) with a select() on HTTPResponse and using its read1(). Because HTTPResponse has a fileno() we're tempted to use select() on it, which works kind-of well... but there is one edge case. While consuming the HTTP response headers, HTTPResponse performs the buffered reading of the response (made by socket.makefile()) which may leave pending buffered payload data. In that case … [View More]

1 0

Non-standard evaluation for Python
by Nima Hamidi Aug. 5, 2019

Aug. 5, 2019

Hello all, I would like to briefly share my thoughts on non-standard evaluation (NSE), why this is useful, and how, potentially, it can be added to Python. In most languages, functions have access only to the *value* of their arguments, and not to the expressions that yielded those values. However, sometimes it is useful to have access to the expressions as well. For example, let plot(x, y, ...) be a function that draws the scatter plot of y against x and adds labels for the x- and y-axes. … [View More]

5 18

Coding environments
by David Mertz Aug. 2, 2019

Aug. 2, 2019

On Fri, Aug 2, 2019, 1:56 PM Chris Angelico <rosuav(a)gmail.com> wrote: > Also a bit old-school (it took me many years to learn the value of > syntax highlighting), and an educator, and I've seen students start > out with Jupyter. As an alternative to the vanilla REPL, I think it's > awesome if a little expensive on low-end machines; but as an > alternative to a text editor, it's an attractive nuisance. Yes, you > can try things out and see the results instantly, AND … [View More]

1 0

PEP: add a `no` keyword as an alias for `not`
by Daniel Okey-Okoro Aug. 2, 2019

Aug. 2, 2019

I think that adding a `no` keyword as an alias for `not` would make for more readable, simple, pythonic code. Take the below: ``` if not val: do_thing_because_value_is_falsy() ``` could be (is actually understood as): ``` if no val: do_thing_because_value_is_falsy() ``` I think this PEP is a work-around for an underlying subtle issue with how the `not` operator is used. It has two use-cases: 1. as a NOT gate for producing opposite boolean values ``` opposite = not regular ``` 2. … [View More]

6 5

Fwd: Re: PEP: add a `no` keyword as an alias for `not`
by Daniel Okey-Okoro Aug. 2, 2019

Aug. 2, 2019

---------- Forwarded message --------- From: Daniel Okey-Okoro <danielokeyokoro(a)gmail.com> Date: Thu, Aug 1, 2019 at 1:37 PM Subject: Re: [Python-ideas] Re: PEP: add a `no` keyword as an alias for `not` To: Chris Angelico <rosuav(a)gmail.com> > not a strong enough justification for breaking any code that uses "no" in any other way. This is a very crucial point I didn't consider. ------------------------------------------------------------ What if we could lexically limit … [View More]

2 1

Fwd: PEP: add a `no` keyword as an alias for `not`
by Daniel Okey-Okoro Aug. 1, 2019

Aug. 1, 2019

---------- Forwarded message --------- From: Daniel Okey-Okoro <danielokeyokoro(a)gmail.com> Date: Thu, Aug 1, 2019 at 1:24 PM Subject: Re: PEP: add a `no` keyword as an alias for `not` To: Calvin Spealman <cspealma(a)redhat.com> Good point Calvin, ------------------------------------------------------------ But in many cases, when people write `if not val` they're checking if the val is `None` i.e. `if val is None`, `if val == None` (so in effect it's kind of like they're … [View More]checking if "something isn't there") I just did a code review. The idea popped in my head after reading code that behaved this way. ------------------------------------------------------------------- Plus, I'm suggesting an alias. In some usecases it would be better to just use `if not val`. i.e. in cases where you are strictly doing True/False assessments not Falsey ( is_empty() ) assessments e.g. None, [], "", 0 ``` Best Intentions, Daniel Okey-Okoro On Thu, Aug 1, 2019, 1:10 PM Calvin Spealman <cspealma(a)redhat.com> wrote: > I think they actually read like they would mean slightly different things, > which would make them existing as aliases confusing. > > I read `if not val` as "If val isn't true" but i would read `if no val` as > "if val does not exist" > > On Thu, Aug 1, 2019 at 4:07 PM Daniel Okey-Okoro < > danielokeyokoro(a)gmail.com> wrote: > >> I think that adding a `no` keyword as an alias for `not` would make for >> more readable, simple, pythonic code. >> >> Take the below: >> >> ``` >> if not val: >> do_thing_because_value_is_falsy() >> ``` >> >> could be (is actually understood as): >> >> ``` >> if no val: >> do_thing_because_value_is_falsy() >> ``` >> >> I think this PEP is a work-around for an underlying subtle issue with how >> the `not` operator is used. >> >> It has two use-cases: >> >> 1. as a NOT gate for producing opposite boolean values >> >> ``` >> opposite = not regular >> ``` >> >> 2. as a sort of ".is_falsy()" checker; when used with an if statement. >> >> like the first example. >> >> >> This PEP would make the difference between the two usecases explicit. >> >> Thoughts? >> >> Best Intentions, >> Daniel Okey-Okoro. >> -- >> https://mail.python.org/mailman/listinfo/python-list >> > > > -- > > CALVIN SPEALMAN > > SENIOR QUALITY ENGINEER > > cspealma(a)redhat.com M: +1.336.210.5107 > [image: https://red.ht/sig] <https://red.ht/sig> > TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> > -- Best Intentions, Daniel Okey-Okoro. [View Less]

1 0

asyncio: futures and tasks with synchronous callbacks
by aurelien.lambert.89＠gmail.com Aug. 1, 2019

Aug. 1, 2019

In asyncio, when a task awaits for another task (or future), it can be cancelled right after the awaited task finished (before the callback have been processed). Thus, if the awaited task has consumed data, the data is lost. For instance, with the following code: import asyncio available_data = [] data_ready = asyncio.Future() def feed_data(data): global data_ready available_data.append(data) data_ready.set_result(None) data_ready = asyncio.… [View More]Future() async def consume_data(): while not available_data: await asyncio.shield(data_ready) return available_data.pop() async def wrapped_consumer(): task = asyncio.ensure_future(consume_data()) return await task If I perform those exact steps: async def test(): task = asyncio.ensure_future(wrapped_consumer()) await asyncio.sleep(0) feed_data('data') await asyncio.sleep(0) task.cancel() await asyncio.sleep(0) print ('task', task) print ('available_data', available_data) loop = asyncio.get_event_loop() loop.run_until_complete(test()) Then I can see that the task has been cancelled despite the data being consumed. Since the result of `wrapped_consumer` cannot be retrieved, the data is forever lost. task <Task cancelled coro=<wrapped_consumer() done, defined at <ipython-input-1-de4ad193b1d0>:17>> available_data [] This side effect does not happen when awaiting a coroutine, but coroutine are not as flexible as tasks. It happens when awaiting a `Future`, a `Task`, or any function like `asyncio.wait`, `asyncio.wait_for` or `asyncio.gather` (which all inherit from or use `Future`). There is then no way to do anything equivalent to: stop_future = asyncio.Future() async def wrapped_consumer2(): task = asyncio.ensure_future(consume_data()) try: await asyncio.wait([task, stop_future]) finally: task.cancel() await asyncio.wait([task]) if not task.cancelled(): return task.result() else: raise RuntimeError('stopped') This is due to the Future calling the callback asynchronously: https://github.com/python/cpython/blob/3.6/Lib/asyncio/futures.py#L214 for callback in callbacks: self._loop.call_soon(callback, self) I propose to create synchronous versions of those, or a `synchronous_callback` parameter, that turns the callbacks of `Future` synchronous. I've experimented a simple librairy `syncio` with CPython 3.6 to do this (it is harder to patch later versions due to the massive use of private methods). Basically, needs to: 1) replace the `Future._schedule_callbacks` method by a synchronous version 2) fix `Task._step` to not fail when cleaning `_current_tasks` (https://github.com/python/cpython/blob/3.6/Lib/asyncio/tasks.py#L245) 3) rewrite all the functions to use synchronous futures instead of normal ones With that librairy, the previous functions are possible and intuitive import syncio async def wrapped_consumer(): task = syncio.ensure_sync_future(consume_data()) return await task stop_future = asyncio.Future() async def wrapped_consumer2(): task = syncio.ensure_sync_future(consume_data()) try: await syncio.sync_wait([task, stop_future]) finally: task.cancel() await asyncio.wait([task]) if not task.cancelled(): return task.result() else: raise RuntimeError('stopped') No need to use `syncio` anywhere else in the code, which makes it totally transparent for the end user. This "library" can be found here: https://github.com/aure-olli/aiokafka/blob/216b4ad3b4115bc9fa463e44fe23636b… It implements `SyncFuture`, `SyncTask`, `ensure_sync_future`, `sync_wait`, `sync_wait_for`, `sync_gather` and `sync_shield`. It works with CPython 3.6 only. To conclude: - asynchronous callbacks are preferrable in most case, but do not provide a coherent cancelled status in specific cases - implementing a version with synchronous callback (or a `synchronous_callback` parameter) is rather easy (however step 2 need to be clarified, probably a cleaner way to fix this) - it is totally transparent for the end user, as synchronous callback are totally compatible with asynchronous ones [View Less]

4 6