Hi Ilya,
I'm not sure that this mailing list (Python-Dev) is the right place for
this discussion, I think that Python-Ideas (CCed) is the correct place.
For the benefit of Python-Ideas, I have left your entire post below, to
establish context.
[Ilya]
> I needed reversed(enumerate(x: list)) in my code, and have discovered
> that it wound't work. This is disappointing because operation is well
> defined.
It isn't really well-defined, since enumerate can operate on infinite
iterators, and you cannot reverse an infinite stream. Consider:
def values():
while True:
yield random.random()
a, b = reversed(enumerate(values())
What should the first pair of (a, b) be?
However, having said that, I think that your idea is not unreasonable.
`enumerate(it)` in the most general case isn't reversable, but if `it`
is reversable and sized, there's no reason why `enumerate(it)` shouldn't
be too.
My personal opinion is that this is a fairly obvious and straightforward
enhancement, one which (hopefully!) shouldn't require much, if any,
debate. I don't think we need a new class for this, I think enhancing
enumerate to be reversable if its underlying iterator is reversable
makes good sense.
But if you can show some concrete use-cases, especially one or two from
the standard library, that would help your case. Or some other languages
which offer this functionality as standard.
On the other hand, I think that there is a fairly lightweight work
around. Define a helper function:
def countdown(n):
while True:
yield n
n -= 1
then call it like this:
# reversed(enumerate(seq))
zip(countdown(len(seq)-1), reversed(seq)))
So it isn't terribly hard to work around this. But I agree that it would
be nice if enumerate encapsulated this for the caller.
One potentially serious question: what should `enumerate.__reversed__`
do when given a starting value?
reversed(enumerate('abc', 1))
Should that yield...?
# treat the start value as a start value
(1, 'c'), (0, 'b'), (-1, 'a')
# treat the start value as an end value
(3, 'c'), (2, 'b'), (1, 'a')
Something else?
My preference would be to treat the starting value as an ending value.
Steven
On Wed, Apr 01, 2020 at 08:45:34PM +0200, Ilya Kamenshchikov wrote:
> Hi,
>
> I needed reversed(enumerate(x: list)) in my code, and have discovered that
> it wound't work. This is disappointing because operation is well defined.
> It is also well defined for str type, range, and - in principle, but not
> yet in practice - on dictionary iterators - keys(), values(), items() as
> dictionaries are ordered now.
> It would also be well defined on any user type implementing __iter__,
> __len__, __reversed__ - think numpy arrays, some pandas dataframes, tensors.
>
> That's plenty of usecases, therefore I guess it would be quite useful to
> avoid hacky / inefficient solutions like described here:
> https://code.activestate.com/lists/python-list/706205/.
>
> If deemed useful, I would be interested in implementing this, maybe
> together with __reversed__ on dict keys, values, items.
>
> Best Regards,
> --
> Ilya Kamen
>
> -----------
> p.s.
>
> *Sketch* of what I am proposing:
>
> class reversible_enumerate:
>
> def __init__(self, iterable):
> self.iterable = iterable
> self.ctr = 0
>
> def __iter__(self):
> for e in self.iterable:
> yield self.ctr, e
> self.ctr += 1
>
> def __reversed__(self):
> try:
> ri = reversed(self.iterable)
> except Exception as e:
> raise Exception(
> "enumerate can only be reversed if iterable to
> enumerate can be reversed and has defined length."
> ) from e
>
> try:
> l = len(self.iterable)
> except Exception as e:
> raise Exception(
> "enumerate can only be reversed if iterable to
> enumerate can be reversed and has defined length."
> ) from e
>
> indexes = range(l-1, -1, -1)
> for i, e in zip(indexes, ri):
> yield i, e
>
> for i, c in reversed(reversible_enumerate("Hello World")):
> print(i, c)
>
> for i, c in reversed(reversible_enumerate([11, 22, 33])):
>
> print(i, c)
> _______________________________________________
> Python-Dev mailing list -- python-dev(a)python.org
> To unsubscribe send an email to python-dev-leave(a)python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NDDKDUD…
> Code of Conduct: http://python.org/psf/codeofconduct/
I know enhancements to pathlib gets brought up occasionally, but it doesn't
look like anyone has been willing to take the initiative and see things
through to completion. I am willing to keep the ball rolling here and even
implement these myself. I have some suggestions and I would like to
discuss them. I don't think any of them are significant enough to require
a pep. These can be split it into independent threads if anyone prefers.
1. copy
The big one people keep bringing up that I strongly agree on is a "copy"
method. This is really the only common file manipulation task that
currently isn't possible. You can make files, read them, move them, delete
them, create directories, even do less common operations like change owners
or create symlinks or hard links.
A common objection is that pathlib doesn't work on multiple paths. But
that isn't the case. There are a ton of methods that do that, including:
* symlink_to
* link_to
* rename
* replace
* glob
* rglob
* iterdir
* is_relative_to
* relative_to
* samefile
I think this is really the only common file operation that someone would
need to switch to a different module to do, and it seems pretty strange to
me to be able to make symbolic or hard links to a file but not straight up
copy one.
2. recursive remove
This could be a "recursive" option to "rmdir" or a "rmtree" method (I
prefer the option). The main reason for this is symmetry. It is possible
to create a tree of folders (using "mkdir(parents=True)"), but once you do
that you cannot remove it again in a straightforward way.
3. newLine for write_text
This is the only relevant option that "Path.open" has but "Path.write_text"
doesn't, and is a serious omission when dealing with multiple operating
systems.
4. uid and gid
You can get the owner and group name of a file (with the "owner" and
"group" methods), but there is no easy way to get the corresponding
number.
5. Stem with no suffixes
The stem property only takes off the last suffix, but even in the example
given ('my/library.tar.gz') it isn't really useful because the suffix has
two parts ('.tar' and '.gz'). I suggest another property, probably
called "rootstem"
or "basestem", that takes off all the suffixes, using the same logic as the
"suffixes" property. This is another symmetry issue: it is possible to
extract all the suffixes, but not remove them.
6. with_suffixes
Equivalent to with_suffix, but replacing all suffixes. Again, this is a
symmetry issue. It is hard to manipulate all the suffixes right now, as
the example show. You can add them or extract them, but not change them
without doing several steps.
7. exist_ok for is_* methods
Currently all the is_* methods (such as is_file) return False if the file
doesn't exist or if it is a broken symlink. This can be dangerous, since
it is not trivially easy to tell if you are dealing with the wrong type of
file vs. a missing file. And it isn't obvious behavior just from the
method name. I suggest adding an "exist_ok" argument to all of these, with
the default being "True" for backwards-compatibility. This argument name
is already in use elsewhere in pathlib. If this is False and the file is
not present, a "FileNotFoundError" is raised.
Consider the following example:
import unittest
def foo():
for x in [1, 2, 'oops', 4]:
print(x + 100)
class TestFoo(unittest.TestCase):
def test_foo(self):
self.assertIs(foo(), None)
if __name__ == '__main__':
unittest.main()
If we were calling `foo` directly we could enter post-mortem debugging via `python -m pdb test.py`.
However since `foo` is wrapped in a test case, `unittest` eats the exception and thus prevents post-mortem debugging. `--failfast` doesn't help, the exception is still swallowed.
Since I am not aware of a solution that enables post-mortem debugging in such a case (without modifying the test scripts, please correct me if one exists), I propose adding a command-line option to `unittest` for [running test cases in debug mode](https://docs.python.org/3/library/unittest.html#unittest.TestCase.deb… so that post-mortem debugging can be used.
P.S.: There is also [this SO question](https://stackoverflow.com/q/4398967/3767239) on a similar topic.
Hi all,
I do not know maybe it was already discussed ... but the toolchain like LLVM is very mature and it can provide the simpler JIT compilation to machine code functionality and it will improve performance of the Python a lot !!
In the current python codebases, there are a lot of decorators with parameters, and their code in most cases contains a lot of code boilerplates. The purpose of this feature is to add `decorator_with_params` (I think the name can be changed to smth better) function to `functools` module. This function will allow using a decorated function as a simple decorator and decorator with parameter.
I believe the real example will bring more context, for instance, `typing._tp_cache` function can be rewritten from:
```
def _tp_cache(func=None, /, *, typed=False):
"""Internal wrapper caching __getitem__ of generic types with a fallback to
original function for non-hashable arguments.
"""
def decorator(func):
cached = functools.lru_cache(typed=typed)(func)
_cleanups.append(cached.cache_clear)
@functools.wraps(func)
def inner(*args, **kwds):
try:
return cached(*args, **kwds)
except TypeError:
pass
return func(*args, **kwds)
return inner
if func is not None:
return decorator(func)
return decorator
```
To
```
@decorator_with_params
def _tp_cache(func, /, *, typed=False):
"""Internal wrapper caching __getitem__ of generic types with a fallback to
original function for non-hashable arguments.
"""
cached = functools.lru_cache(typed=typed)(func)
_cleanups.append(cached.cache_clear)
@functools.wraps(func)
def inner(*args, **kwds):
try:
return cached(*args, **kwds)
except TypeError:
pass # All real errors (not unhashable args) are raised below.
return func(*args, **kwds)
return inner
```
And `_tp_cache` can be used as:
```
@_tp_cache(typed=True)
def __getitem__(self, parameters):
return self._getitem(self, parameters)
...
@_tp_cache
def __getitem__(self, parameters):
return self._getitem(self, parameters)
```
Proposed implementation is quite simple (I used it in a lot of projects):
```
def decorator_with_params(deco):
@wraps(deco)
def decorator(func=None, /, *args, **kwargs):
if func is not None:
return deco(func, *args, **kwargs)
def wrapper(f):
return deco(f, *args, **kwargs)
return wrapper
return decorator
```
I believe it will be a great enhancement for the Python standard library and will save such important indentations.
I have already opened an issue https://bugs.python.org/issue42455 (Sorry about that I didn't know that smth like python-ideas exists).
There's a whole matrix of these and I'm wondering why the matrix is
currently sparse rather than implementing them all. Or rather, why we
can't stack them as:
class foo(object):
@classmethod
@property
def bar(cls, ...):
...
Essentially the permutation are, I think:
{'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable
attribute}.
concreteness
implicit first arg
type
name
comments
{unadorned}
{unadorned}
method
def foo():
exists now
{unadorned} {unadorned} property
@property
exists now
{unadorned} {unadorned} non-callable attribute
x = 2
exists now
{unadorned} static
method @staticmethod
exists now
{unadorned} static property @staticproperty
proposing
{unadorned} static non-callable attribute {degenerate case -
variables don't have arguments}
unnecessary
{unadorned} class
method @classmethod
exists now
{unadorned} class property @classproperty or @classmethod;@property
proposing
{unadorned} class non-callable attribute {degenerate case - variables
don't have arguments}
unnecessary
abc.abstract {unadorned} method @abc.abstractmethod
exists now
abc.abstract {unadorned} property @abc.abstractproperty
exists now
abc.abstract {unadorned} non-callable attribute
@abc.abstractattribute or @abc.abstract;@attribute
proposing
abc.abstract static method @abc.abstractstaticmethod
exists now
abc.abstract static property @abc.staticproperty
proposing
abc.abstract static non-callable attribute {degenerate case -
variables don't have arguments} unnecessary
abc.abstract class method @abc.abstractclassmethod
exists now
abc.abstract class property @abc.abstractclassproperty
proposing
abc.abstract class non-callable attribute {degenerate case -
variables don't have arguments} unnecessary
I think the meanings of the new ones are pretty straightforward, but in
case they are not...
@staticproperty - like @property only without an implicit first
argument. Allows the property to be called directly from the class
without requiring a throw-away instance.
@classproperty - like @property, only the implicit first argument to the
method is the class. Allows the property to be called directly from the
class without requiring a throw-away instance.
@abc.abstractattribute - a simple, non-callable variable that must be
overridden in subclasses
@abc.abstractstaticproperty - like @abc.abstractproperty only for
@staticproperty
@abc.abstractclassproperty - like @abc.abstractproperty only for
@classproperty
--rich
Hello everyone,
We have many compression algorithms as standard modules, but we lack two important new algorithms: brotli and zstandard.
Brotli is an important compression algorithm both for HTTP2 and for compressing/decompressing HTTP responses.
ZStandard is another algorithm for compression that can easily beat gzip with a better compression ratio and less compression time.
Both are now stable enough and are past the 1.0 mark.
Let's include them in the standard library and, of course, make them optional at build time like the rest of our compression modules.
What do you think?
# Experimental Syntax Proposal
I would like to propose that Python adopts a modified process
before introducing significant changes of its syntax.
## Preamble
Given the following file:
```python
# coding: experimental-syntax
from experimental-syntax import fraction_literal
from experimental-syntax import decimal_literal
assert 1 /3F == Fraction(1, 3)
assert 0.33D == Decimal('0.33')
print("simple_test.py ran successfully.")
```
This is what happens when I run it, with the standard Python interpreter:
```
$ python simple_test.py
simple_test.py ran successfully.
```
In what follows, I use this as a concrete example for one
of the three possible options mentioned.
## The problem
Python evolves in many ways including with the addition
of new modules in the standard library and
with the introduction of new syntax.
Before considering the addition of a module in
the standard library, it is often suggested to have a
version on Pypi so that users can experiment with it,
which can lead to significant improvements.
However, when it comes to proposed syntax changes,
this is currently not possible to do, at least not in any
standard way that would immediately be easily recognized
as such by the wider community.
## Proposed solutions
For those that agree that this is something that could be
improved upon, I see at least three possible solutions.
1. Adoption of a simple convention to identify the use of non-standard
syntax.
This could take the form of a comment with a specific form
introduced near the top of the file. This comment could be used as
a search term, to identify modules or projects that make use of
some custom syntax.
2. Addition of a special encoding module in the Python standard library
that can be used to implement nonstandard syntax through source
transformation. The short program written at the top provides an
example of what this might look like. Note the comment at the top
defining a special codec, which could also
3. Addition of a standard import hook in the standard library which could
be used, like the special encoding example above, to implement
syntactic changes.
By doing searches in standard locations (Github, Gitlab, etc.) on, one could
identify how many modules are making use of a given experimental syntax,
giving an idea of the desirability of incorporating into Python.
Imagine how different the discussion about the walrus operator would have
been
if people had had the opportunity to try it out on their own for a few
months
prior to a decision being made about adding it to Python.
## New Syntax: two approaches
There are currently at least two ways in which one can write a Python
program
that can use some non standard syntax:
* By using an import hook. This is the most powerful approach as it
allows one to do changes either at the source level
(prior to parsing), or at the AST level, or both.
* By using a custom encoding. This only allows transformations at the
source level.
Both approaches currently require a two-step process.
To use an import hook, one has first to load a function that sets up this
import hook before importing the module that contains the experimental
syntax. This makes it impossible to simply write
```
python module_with_new_syntax.py
```
and have it being executed. Import hooks, like the name implies,
only apply to modules being imported - not to the first module executed by
Python.
With a custom encoding, it is however possible to first register a
custom codec either via a site.py or usercustomize.py file.
Once this is done, the above way of running a module with new syntax is
possible,
provided that the appropriate encoding declaration is included in the
script.
This is what I did with the example shown above.
### Current limitation of these two approaches
Using an import hook for enabling new syntax does not work
for code entered in the standard Python REPL, which limits the possibility
of experimentation.
While I have read that specifying the environment
variable `PYTHONIOENCODING` enables the Python REPL to use a custom
encoding,
I have not been able to make it work on Windows and
confirm that it is indeed possible to do so.
However, I have been able to use a simple customized REPL that can make use
of the above. Perhaps the standard REPL could be modified in a similar way.
## New suggested process
Assuming one of the options mentioned above is adopted,
before changes to the syntax are introduced in a Python version,
they would be made available to be used either as an encoding variant
or an import hook giving enough time for interested pythonistas to
experiment with the new syntax, writing actual code and possibly
trying out various alternatives.
## Proof of concept
As a proof of concept shown at the beginning of this post,
I have created two tiny modules that introduce new syntax that
had been discussed many times on this list:
1. a new syntax for decimal literals
2. a new syntax for rationals
Both modules have been uploaded separately to Pypi;
I wanted to simulate what could happen if a proposal such
as this one were adopted.
To make use of these, you need to use the `experimental-syntax` codec
found in my `ideas` package.
To run the example as shown above, you first need to register the codec,
which can be done by using either the `site.py` or `usercustomize.py`
approach.
I chose the latter, by setting the environment variable `PYTHONPATH` to be
a path where the following `usercustomize.py` file is found.
```python
from ideas import experimental_syntax_encoding
print(f" --> {__file__} was executed")
```
Doing this on Windows, I found that it did not seem to work when
using a virtual environment (I added the print statement to confirm
that it was loaded).
Here's a sample session using the code available currently on pypi.
```
C:\Users\andre\github\ideas>python simple_test.py
--> C:\Users\andre\github\ideas\usercustomize.py was executed
simple_test.py ran successfully.
C:\Users\andre\github\ideas>python
--> C:\Users\andre\github\ideas\usercustomize.py was executed
Python 3.8.4 (tags/v3.8.4:dfa645a, Jul 13 2020, 16:30:28) [MSC v.1926 32
bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from ideas import experimental_syntax_encoding
>>> from ideas import console
>>> console.start()
Configuration values for the console:
transform_source: <function transform_source at 0x00A8AE38>
--------------------------------------------------
Ideas Console version 0.0.19. [Python version: 3.8.4]
~>> import simple_test
simple_test.py ran successfully.
~>> from experimental-syntax import decimal_literal
~>> 3.46D
Decimal('3.46')
~>> from experimental-syntax import fraction_literal
~>> 2/3F
Fraction(2, 3)
~>>
```
Installation:
python -m pip install ideas # will also install token-utils
python -m pip install decimal-literal
python -m pip install fraction-literal
More information about "ideas" can be found at
https://aroberge.github.io/ideas/docs/html/
It has not (yet) been updated to include anything about the
`experimental_syntax_encoding` module, which I just
worked on today.
## Final thoughts
This proposal is really about the idea of adopting a standard process of
some
sort that enables users to experiment with any proposed new syntax,
rather than the specific "silly" examples I have chosen to illustrate it.
André Roberge
P.S. If anyone knows how to make a usercustomize file run in a virtual
environment on Windows, please let me know.
Starting a new thread for this as suggested by Guido.
On 18/11/20 7:20 pm, Guido van Rossum wrote:
> On Tue, Nov 17, 2020 at 22:12 Greg Ewing <greg.ewing(a)canterbury.ac.nz
> <mailto:greg.ewing@canterbury.ac.nz>> wrote:
>
> If there's anything I would change, it would be to have the for
> statement create a new binding on each iteration, so that capturing
> it with a def or lambda inside the loop works as expected. I even
> came up with a way to do that while still allowing the last-bound
> value to be seen afterwards, but the idea didn't gain any traction.
>
> I wonder if we should try that idea again? The problem may be backwards
> compatibility — there’s always someone who depends on such a thing (even
> if by accident, it would be a pain to debug). Maybe we could invent new
> syntax to signal this behavior, though that might defeat the purpose.
I think it can be made close enough to backwards compatible, although
I'm not greatly opposed to new syntax if it's thought desirable.
Essentially the idea is this: If the loop body contains a def or lambda
that captures the loop variable, then on each iteration, a new binding
is created, instead of changing an existing binding.
Note that this is *not* the same as introducing a new scope. All the
bindings share the same name, and from the perspective of code outside
the nested function, nothing changes -- the name is still bound to
the most recent value, and can still be accessed after the loop has
finished.
In CPython, it would be implemented by creating a new cell for each
iteration, instead of changing the value of an existing cell.
For other implementations, I don't know -- it would depend on how
those implementations currently deal with nested functions.
Do we need new syntax? My feeling is, probably not. I believe that
the vast majority of existing code, perhaps all of it, would be
unaffected by this change. All the existing workarounds for the current
behaviour would continue to work. The only things that would break
would be code that somehow relies on capturing the same instance of
the loop variable every time, and I find it hard to imagine code that
needs to do that.
If we do want new syntax, my suggestion would be
for new x in some_iterator:
...
The downside of requiring special syntax is that we would still
regularly get people asking why their lambdas in for statements don't
do what they expect. We would have a better answer for them, but the
questions wouldn't go away.
--
Greg
Instead of importing “Any" from the typing module, we can annotate our functions with “Any" right away without the extra step. What do you think? We have the builtin function “any” which some Python users could mistakingly use, but static type checkers should catch that.