Why do i need to convince a core developer for my PEP? AFAIK the steering
council can include non core developers (i know it isn't that current case
but for the future this is important). And if the last authority who will
approve my PEP is the steering council i just need to convince them not
core developers.
Sponsors can stay (and they should because guidance is important) but thy
shouldn't be mandatory. Let everyone to send their peps.
Good day all,
as a continuation of thread "OS related file operations (copy, move,
delete, rename...) should be placed into one module"
https://mail.python.org/pipermail/python-ideas/2017-January/044217.html
please consider making pathlib to a central file system module with putting
file operations (copy, move, delete, rmtree etc) into pathlib.
BR,
George
> On Friday, April 6, 2018 at 8:14:30 AM UTC-7, Guido van Rossum wrote:
> On Fri, Apr 6, 2018 at 7:47 AM, Peter O'Connor <peter.ed...(a)gmail.com> wrote:
>> So some more humble proposals would be:
>>
>> 1) An initializer to itertools.accumulate
>> functools.reduce already has an initializer, I can't see any controversy to adding an initializer to itertools.accumulate
>
> See if that's accepted in the bug tracker.
It did come-up once but was closed for a number reasons including lack of use cases. However, Peter's signal processing example does sound interesting, so we could re-open the discussion.
For those who want to think through the pluses and minuses, I've put together a Q&A as food for thought (see below). Everybody's design instincts are different -- I'm curious what you all think think about the proposal.
Raymond
---------------------------------------------
Q. Can it be done?
A. Yes, it wouldn't be hard.
_sentinel = object()
def accumulate(iterable, func=operator.add, start=_sentinel):
it = iter(iterable)
if start is _sentinel:
try:
total = next(it)
except StopIteration:
return
else:
total = start
yield total
for element in it:
total = func(total, element)
yield total
Q. Do other languages do it?
A. Numpy, no. R, no. APL, no. Mathematica, no. Haskell, yes.
* http://docs.scipy.org/doc/numpy/reference/generated/numpy.ufunc.accumulat...
* https://stat.ethz.ch/R-manual/R-devel/library/base/html/cumsum.html
* http://microapl.com/apl/apl_concepts_chapter5.html
\+ 1 2 3 4 5
1 3 6 10 15
* https://reference.wolfram.com/language/ref/Accumulate.html
* https://www.haskell.org/hoogle/?hoogle=mapAccumL
Q. How much work for a person to do it currently?
A. Almost zero effort to write a simple helper function:
myaccum = lambda it, func, start: accumulate(chain([start], it), func)
Q. How common is the need?
A. Rare.
Q. Which would be better, a simple for-loop or a customized itertool?
A. The itertool is shorter but more opaque (especially with respect
to the argument order for the function call):
result = [start]
for x in iterable:
y = func(result[-1], x)
result.append(y)
versus:
result = list(accumulate(iterable, func, start=start))
Q. How readable is the proposed code?
A. Look at the following code and ask yourself what it does:
accumulate(range(4, 6), operator.mul, start=6)
Now test your understanding:
How many values are emitted?
What is the first value emitted?
Are the two sixes related?
What is this code trying to accomplish?
Q. Are there potential surprises or oddities?
A. Is it readily apparent which of assertions will succeed?
a1 = sum(range(10))
a2 = sum(range(10), 0)
assert a1 == a2
a3 = functools.reduce(operator.add, range(10))
a4 = functools.reduce(operator.add, range(10), 0)
assert a3 == a4
a4 = list(accumulate(range(10), operator.add))
a5 = list(accumulate(range(10), operator.add, start=0))
assert a5 == a6
Q. What did the Python 3.0 Whatsnew document have to say about reduce()?
A. "Removed reduce(). Use functools.reduce() if you really need it; however, 99 percent of the time an explicit for loop is more readable."
Q. What would this look like in real code?
A. We have almost no real-world examples, but here is one from a StackExchange post:
def wsieve(): # wheel-sieve, by Will Ness. ideone.com/mqO25A->0hIE89
wh11 = [ 2,4,2,4,6,2,6,4,2,4,6,6, 2,6,4,2,6,4,6,8,4,2,4,2,
4,8,6,4,6,2,4,6,2,6,6,4, 2,4,6,2,6,4,2,4,2,10,2,10]
cs = accumulate(cycle(wh11), start=11)
yield( next( cs)) # cf. ideone.com/WFv4f
ps = wsieve() # codereview.stackexchange.com/q/92365/9064
p = next(ps) # 11
psq = p*p # 121
D = dict( zip( accumulate(wh11, start=0), count(0))) # start from
sieve = {}
for c in cs:
if c in sieve:
wheel = sieve.pop(c)
for m in wheel:
if not m in sieve:
break
sieve[m] = wheel # sieve[143] = wheel@187
elif c < psq:
yield c
else: # (c==psq)
# map (p*) (roll wh from p) = roll (wh*p) from (p*p)
x = [p*d for d in wh11]
i = D[ (p-11) % 210]
wheel = accumulate(cycle(x[i:] + x[:i]), start=psq)
p = next(ps) ; psq = p*p
next(wheel) ; m = next(wheel)
sieve[m] = wheel
I'm parsing configs for domain filtering rules, and they come as a list.
However, str.endswith requires a tuple. So I need to use
str.endswith(tuple(list)). I don't know the reasoning for this, but why
not just accept a list as well?
From the description of the import statement in the language reference [1]:
> The *public names* defined by a module are determined by checking the module's
> namespace for a variable named ``__all__``; if defined, it must be a sequence
> of strings which are names defined or imported by that module. The names
> given in ``__all__`` are all considered public and are required to exist. If
> ``__all__`` is not defined, the set of public names includes all names found
> in the module's namespace which do not begin with an underscore character
> (``'_'``). ``__all__`` should contain the entire public API. It is intended
> to avoid accidentally exporting items that are not part of the API (such as
> library modules which were imported and used within the module).
As an example of names which should be excluded from the public API the
reference mentions names of modules which were imported and used within
the module. And they are perhaps the most common cases. It is often
imported module names are the only non-underscored names which are
imported by a star-import by accident.
For example, modules re and sys were used in Lib/idlelib/editor.py but
there were no imports for them. This worked because they were imported
implicitly by `from tkinter import *`. [1]
There is other case when a module name can be occurred in the module
namespace: if it is a submodule. For example:
>>> from xml.etree import *
>>> ElementTree
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'ElementTree' is not defined
>>> ElementPath
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'ElementPath' is not defined
>>> import xml.etree.ElementTree
>>> from xml.etree import *
>>> ElementTree
<module 'xml.etree.ElementTree' from
'/home/serhiy/py/cpython3.8/Lib/xml/etree/ElementTree.py'>
>>> ElementPath
<module 'xml.etree.ElementPath' from
'/home/serhiy/py/cpython3.8/Lib/xml/etree/ElementPath.py'>
Names imported by a `from xml.etree import *` are depended on importing
a xml.etree's submodule ElementTree. I do not think this is good.
I propose to change the rule for determining the set of public names if
`__all__` is not defined. In addition to underscored names I propose to
exclude names of modules.
In these rare cases where modules should be imported in a star import it
is not difficult to introduce `__all__`.
[1]
https://docs.python.org/3/reference/simple_stmts.html#the-import-statement
[2] https://bugs.python.org/issue29446
Hello everybody,
Scala 3 propose the a new syntax for Union type. See here
<https://dotty.epfl.ch/docs/reference/new-types/union-types.html>. I
propose to add a similar syntax in Python.
# Operator for Union
assert( int | str == Union[int,str])
assert( int | str | float == Union[int,str,float])
# Operator for Optional
assert( ~int == Optional[int])
Now, it's possible to write:
def fn(bag:List[int | str], option: ~int = None) -> float | str: ...
in place of
def fn(bag:List[Option[int,str]], option: Optional[int] = None) ->
Union[float,str]: ...
I think these syntaxes are more clear, and can help with the adoption of
typing.
I test and implement these ideas in a two fork : One for CPython
<https://github.com/pprados/cpython> and one for MyPy
<https://github.com/pprados/mypy>. See the branches add_OR_to_types (for
Union syntax) or add_INVERT_to_types (for Union and Optional syntax).
How I implement that ? I add the operators __or__ and __revert__ to
PyType_Type. The C code is similar of :
from typing import *
def type_or(self,right):
return Union[self,right]
type(type).__or__ = type_or
Actually, the accepted syntax for typing is :
annotation: name_type
name_type: NAME (args)?
args: '[' paramslist ']'
paramslist: annotation (',' annotation)* [',']
I propose to extend the syntax to :
annotation: ( name_type | or_type | invert_type )
name_type: NAME (args)?
args: '[' paramslist ']'
paramslist: annotation (',' annotation)* [',']
or_type: name_type '|' annotation
invert_type: '~' annotation
What do you think about that ?
The draft of a PEP is here
<https://github.com/pprados/peps/blob/master/pep-9999.rst>.
Regards
I did some searching to see if this has already been proposed before, but didn't find any evidence that it has. If so, let me know and I'll go away :)
One of the tasks that I encounter frequently enough is retrieving a nested key from a nested collection of dictionaries (a dictionary of dictionaries that can be any number of layers deep). There are multiple ways of tackling this, normally done with `reduce()`, iterative looping or chained `get()`s and then handling `None`, `KeyError` or `AttributeError` appropriately. I'd like to avoid this extra code and logic and have a built-in method to facilitate this common data access pattern.
I'd like to propose that we add a `dig()` method to dictionaries analogous to Ruby's `dig()` method for Ruby Hashes (reference: https://ruby-doc.org/core-2.6.0.preview2/Hash.html#method-i-dig). Initially, I'd suggest that we only support nested dictionaries, but we could also support lists and other collection types as Ruby does if we really want to. Similar to the existing `get()` method on dictionaries, I'd propose that the method return `None` if any of the keys in the chain is not found, avoiding `KeyError`.
Thoughts?
In Python strings are allowed to have a number of special prefixes:
b'', r'', u'', f''
+ their combinations.
The proposal is to allow arbitrary (or letter-only) user-defined prefixes as well.
Essentially, a string prefix would serve as a decorator for a string, allowing the
user to impose a special semantics of their choosing.
There are quite a few situations where this can be used:
- Fraction literals: `frac'123/4567'`
- Decimals: `dec'5.34'`
- Date/time constants: `t'2019-08-26'`
- SQL expressions: `sql'SELECT * FROM tbl WHERE a=?'.bind(a=...)`
- Regular expressions: `rx'[a-zA-Z]+'`
- Version strings: `v'1.13.0a'`
- etc.
This proposal has been already discussed before, in 2013:
https://mail.python.org/archives/list/python-ideas@python.org/thread/M3OL...
The opinions were divided whether this is a useful addition. The opponents
mainly argued that as this only "saves a couple of keystrokes", there is no
need to overcomplicate the language. It seems to me that now, 6 years later,
that argument can be dismissed by the fact that we had, in fact, added new
prefix "f" to the language. Note how the "format strings" would fall squarely
within this framework if they were not added by now.
In addition, I believe that "saving a few keystroked" is a worthy goal if it adds
considerable clarity to the expression. Readability counts. Compare:
v"1.13.0a"
v("1.13.0a")
To me, the former expression is far easier to read. Parentheses, especially as
they become deeply nested, are not easy on the eyes. But, even more importantly,
the first expression much better conveys the *intent* of a version string. It has
a feeling of an immutable object. In the second case the string is passed to the
constructor, but the string has no meaning of its own. As such, the second
expression feels artificial. Consider this: if the feature already existed, how *would*
you prefer to write your code?
The prefixes would also help when writing functions that accept different types
of their argument. For example:
collection.select("abc") # find items with name 'abc'
collection.select(rx"[abc]+") # find items that match regular expression
I'm not discussing possible implementation of this feature just yet, we can get to
that point later when there is a general understanding that this is worth considering.
Goal - using wheels rather than RPM and/or installp formats for
distributing binary modules
Why bother?
One reason (there are likely more) - using wheels means packages/modules
can be loaded in a virtualenv
rather than require they are first loaded in the system environment
using installp/rpm/yum (with root authority). This alone has been reason
enough for me to do the research.
Introduction
The use of pip and wheels is commonplace in the worlds of Linux, macOS
and Windows.
Not so for AIX. Not because it couldn't be commonplace.
The current situation for AIX is comparable to the initial issue Linux
was facing
when PEP513 was written:
"Currently, distribution of binary Python extensions for Windows and OS
X is straightforward. ...
For Linux, the situation is much more delicate. ...
Build tools using PEP 425 platform tags [3] do not track information
about the particular Linux distribution or installed system libraries,
and instead assign all wheels the too-vague linux_i686 or linux_x86_64
tags. Because of this ambiguity, ..."
The root cause for the *ambiguity* that Linux systems had is not the
ambiguity that AIX faces.
AIX has provided a consistent way to "tag" it's runtime environment
since at least 2007 (AIX 5.3 TL7).
Since that time IBM AIX has also *guaranteed* binary compatibility for
migration of applications
from old to new OS levels.
I would like to see these tags added - at a minimum that they can be
retrieved by something such as sysconfig.get_var('AIC_BLD_TAG'). It
would be "nice" to see sysconfig.get_platform() updated to include these
values from the running system.
Further, while pip related tools can add the "bitness" to the platform
tags I would like to see something added to the AIX get_platform() tag
(b32, ppc, aix32), (b64, ppc64, aix64) for 32 and 64 bit operations,
respectively - as that is a "running" environment attribute. Open to
other ideas on what the bitness tag should be. IMHO - anything is better
than nothing. Maybe this could be considered a bug rather than as a new
feature.
Thank you for your feedback.
Michael
I have found myself in an awkward situation with current (Python 3.7) JSON module. Basically it boils down to how it handles floats. I had been hit on this particular case:
In [31]: float(0.6441726684570313)
Out[31]: 0.6441726684570312
but I guess it really does not matter.
What matters is that I did not find a way how to fix it with the standard `json` module. I have the JSON file generated by another program (C++ code, which uses nlohmann/json library), which serializes one of the floats to the value above. Then when reading this JSON file in my Python code, I can get either decimal.Decimal object (when specifying `parse_float=decimal.Decimal`) or float. If I use the latter the least significant digit is lost in deserialization.
If I use Decimal, the value is preserved, but there seems to be no way to "serialize it back". Writing a custom serializer:
class DecimalEncoder(json.JSONEncoder):
def default(self, o):
if isinstance(o, decimal.Decimal):
return str(o) # <- This becomes quoted in the serialized output
return super.default(o)
seems to only allow returning "string" value, but then serializes it as a string! I.e. with the double quotes. What seems to be missing is an ability to return a "raw textual representation" of the serialized object which will not get mangled further by the `json` module.
I noticed that `simplejson` provides an explicit option for its standard serializing function, called `use_decimal`, which basically solves my problem., but I would just like to use the standard module, I guess.
So the question is, if something like `use_decimal` has been considered for the standard module, and if yes, why it was not implemented, or the other option could be to support "raw output" in the serializer, e.g. something like:
class DecimalEncoder(json.JSONEncoder):
def raw(self, o):
if isinstance(o, decimal.Decimal):
return str(o) # <- This is a raw representation of the object
return super.raw(o)
Where the returning values will be directly passed to the output stream without adding any additional characters. Then I could write my own Decimal serializer with few lines of code above.
If anyone would want to know, why the last digit matters (or why I cannot double quote the floats), it is because the file has a secure hash attached and this basically breaks it.
