Mailman 3 May 2020 - Python-ideas

zip(x, y, z, strict=True)
by Ram Rachum Dec. 1, 2022

Dec. 1, 2022

Here's something that would have saved me some debugging yesterday: >>> zipped = zip(x, y, z, strict=True) I suggest that `strict=True` would ensure that all the iterables have been exhausted, raising an exception otherwise. This is useful in cases where you're assuming that the iterables all have the same lengths. When your assumption is wrong, you currently just get a shorter result, and it could take you a while to figure out why it's happening. What do you think?

27 179

Adding a thin wrapper class around the functions in stdlib.heapq
by bunslow Feb. 17, 2022

Feb. 17, 2022

Nothing so bombastic this time. The heapq functions are basically all named "heapsomething", and basically all take a "heap" for their first argument, with supplementary args coming after. It's a textbook example of the (hypothetical) Object Oriented Manifesto™ where defining a class increases type safety and programmers' conceptual clarity. There're practically no drawbacks, and the code to be added would be very simple. Updating the tests and docs would probably be harder. In pure Python, … [View More]

17 32

Make fnmatch.filter accept a tuple of patterns
by Andre Delfino March 26, 2021

March 26, 2021

Frequently, while globbing, one needs to work with multiple extensions. I’d like to propose for fnmatch.filter to handle a tuple of patterns (while preserving the single str argument functionality, alas str.endswith), as a first step for glob.i?glob to accept multiple patterns as well. Here is the implementation I came up with: https://github.com/python/cpython/compare/master...andresdelfino:fnmatch-mu… If this is deemed reasonable, I’ll write tests and documentation updates. Any opinion?

9 15

Dataclasses, keyword args, and inheritance
by George Leslie-Waksman March 11, 2021

March 11, 2021

The proposed implementation of dataclasses prevents defining fields with defaults before fields without defaults. This can create limitations on logical grouping of fields and on inheritance. Take, for example, the case: @dataclass class Foo: some_default: dict = field(default_factory=dict) @dataclass class Bar(Foo): other_field: int this results in the error: 5 @dataclass ----> 6 class Bar(Foo): 7 other_field: int 8 ~/.pyenv/versions/3.6.2/envs/… [View More]clover_pipeline/lib/python3.6/site-packages/dataclasses.py in dataclass(_cls, init, repr, eq, order, hash, frozen) 751 752 # We're called as @dataclass, with a class. --> 753 return wrap(_cls) 754 755 ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in wrap(cls) 743 744 def wrap(cls): --> 745 return _process_class(cls, repr, eq, order, hash, init, frozen) 746 747 # See if we're being called as @dataclass or @dataclass(). ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in _process_class(cls, repr, eq, order, hash, init, frozen) 675 # in __init__. Use "self" if possible. 676 '__dataclass_self__' if 'self' in fields --> 677 else 'self', 678 )) 679 if repr: ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in _init_fn(fields, frozen, has_post_init, self_name) 422 seen_default = True 423 elif seen_default: --> 424 raise TypeError(f'non-default argument {f.name!r} ' 425 'follows default argument') 426 TypeError: non-default argument 'other_field' follows default argument I understand that this is a limitation of positional arguments because the effective __init__ signature is: def __init__(self, some_default: dict = <something>, other_field: int): However, keyword only arguments allow an entirely reasonable solution to this problem: def __init__(self, *, some_default: dict = <something>, other_field: int): And have the added benefit of making the fields in the __init__ call entirely explicit. So, I propose the addition of a keyword_only flag to the @dataclass decorator that renders the __init__ method using keyword only arguments: @dataclass(keyword_only=True) class Bar(Foo): other_field: int --George Leslie-Waksman [View Less]

12 35

Add extend_const action to argparse
by python＠roganartu.com Feb. 7, 2021

Feb. 7, 2021

Forgive me if this idea has been discussed before, I searched the mailing lists, the CPython repo, and the issue tracker and was unable to find anything. I have found myself a few times in a position where I have a repeated argument that uses the `append` action, along with some convenience arguments that append a specific const to that same dest (eg: `--filter-x` being made equivalent to `--filter x` via `append_const`). This is particularly useful in cli apps that expose some kind of … [View More]

3 2

Re: [Python-Dev] reversed enumerate
by Steven D'Aprano Jan. 21, 2021

Jan. 21, 2021

Hi Ilya, I'm not sure that this mailing list (Python-Dev) is the right place for this discussion, I think that Python-Ideas (CCed) is the correct place. For the benefit of Python-Ideas, I have left your entire post below, to establish context. [Ilya] > I needed reversed(enumerate(x: list)) in my code, and have discovered > that it wound't work. This is disappointing because operation is well > defined. It isn't really well-defined, since enumerate can operate on infinite … [View More]iterators, and you cannot reverse an infinite stream. Consider: def values(): while True: yield random.random() a, b = reversed(enumerate(values()) What should the first pair of (a, b) be? However, having said that, I think that your idea is not unreasonable. `enumerate(it)` in the most general case isn't reversable, but if `it` is reversable and sized, there's no reason why `enumerate(it)` shouldn't be too. My personal opinion is that this is a fairly obvious and straightforward enhancement, one which (hopefully!) shouldn't require much, if any, debate. I don't think we need a new class for this, I think enhancing enumerate to be reversable if its underlying iterator is reversable makes good sense. But if you can show some concrete use-cases, especially one or two from the standard library, that would help your case. Or some other languages which offer this functionality as standard. On the other hand, I think that there is a fairly lightweight work around. Define a helper function: def countdown(n): while True: yield n n -= 1 then call it like this: # reversed(enumerate(seq)) zip(countdown(len(seq)-1), reversed(seq))) So it isn't terribly hard to work around this. But I agree that it would be nice if enumerate encapsulated this for the caller. One potentially serious question: what should `enumerate.__reversed__` do when given a starting value? reversed(enumerate('abc', 1)) Should that yield...? # treat the start value as a start value (1, 'c'), (0, 'b'), (-1, 'a') # treat the start value as an end value (3, 'c'), (2, 'b'), (1, 'a') Something else? My preference would be to treat the starting value as an ending value. Steven On Wed, Apr 01, 2020 at 08:45:34PM +0200, Ilya Kamenshchikov wrote: > Hi, > > I needed reversed(enumerate(x: list)) in my code, and have discovered that > it wound't work. This is disappointing because operation is well defined. > It is also well defined for str type, range, and - in principle, but not > yet in practice - on dictionary iterators - keys(), values(), items() as > dictionaries are ordered now. > It would also be well defined on any user type implementing __iter__, > __len__, __reversed__ - think numpy arrays, some pandas dataframes, tensors. > > That's plenty of usecases, therefore I guess it would be quite useful to > avoid hacky / inefficient solutions like described here: > https://code.activestate.com/lists/python-list/706205/. > > If deemed useful, I would be interested in implementing this, maybe > together with __reversed__ on dict keys, values, items. > > Best Regards, > -- > Ilya Kamen > > ----------- > p.s. > > *Sketch* of what I am proposing: > > class reversible_enumerate: > > def __init__(self, iterable): > self.iterable = iterable > self.ctr = 0 > > def __iter__(self): > for e in self.iterable: > yield self.ctr, e > self.ctr += 1 > > def __reversed__(self): > try: > ri = reversed(self.iterable) > except Exception as e: > raise Exception( > "enumerate can only be reversed if iterable to > enumerate can be reversed and has defined length." > ) from e > > try: > l = len(self.iterable) > except Exception as e: > raise Exception( > "enumerate can only be reversed if iterable to > enumerate can be reversed and has defined length." > ) from e > > indexes = range(l-1, -1, -1) > for i, e in zip(indexes, ri): > yield i, e > > for i, c in reversed(reversible_enumerate("Hello World")): > print(i, c) > > for i, c in reversed(reversible_enumerate([11, 22, 33])): > > print(i, c) > _______________________________________________ > Python-Dev mailing list -- python-dev(a)python.org > To unsubscribe send an email to python-dev-leave(a)python.org > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NDDKDUD… > Code of Conduct: http://python.org/psf/codeofconduct/ [View Less]

7 14

Add command-line option to unittest for enabling post-mortem debugging
by Dominik Vilsmeier Jan. 9, 2021

Jan. 9, 2021

Consider the following example: import unittest def foo(): for x in [1, 2, 'oops', 4]: print(x + 100) class TestFoo(unittest.TestCase): def test_foo(self): self.assertIs(foo(), None) if __name__ == '__main__': unittest.main() If we were calling `foo` directly we could enter post-mortem debugging via `python -m pdb test.py`. However since `foo` is wrapped in a test case, `unittest` eats the exception and thus prevents post-… [View More]

1 1

Python JIT Compilation Thoughts
by redradist＠gmail.com Jan. 5, 2021

Jan. 5, 2021

Hi all, I do not know maybe it was already discussed ... but the toolchain like LLVM is very mature and it can provide the simpler JIT compilation to machine code functionality and it will improve performance of the Python a lot !!

9 15

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley Dec. 16, 2020

Dec. 16, 2020

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists … [View More]now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich [View Less]

10 15

Compound with .. else statement
by Jimmy Thrasibule Sept. 9, 2020

Sept. 9, 2020

Hi, In Python, there are multiple [compound statements](https://docs.python.org/3/reference/compound_stmts.html) with the `else` keyword. For example: ``` for x in iterable: if x == sentinel: break else: print("Sentinel not found.") ``` or: ``` try: do_something_sensitive() except MyError: print("Oops!") else: print("We're all safe.") ``` In my situation, I would like to mix the `with` statement with `else`. In this case I would like that if no exception is … [View More]

6 10

Aside RE: Re: macaddress or networkaddress module
by Steve Barnes Aug. 13, 2020

Aug. 13, 2020

As an aside I have a perfect example to back up what Paul is saying below. I work for a large corporation where developers are permitted to install python modules on their development machines, (subject to some licence restrictions), but the proxy settings required to get to PyPi vary from user to user, site to site, day to day (sometimes hour to hour) and connection type to connection type, e.g. If you are connected via WiFi you need to use a different proxy to that needed if you are on a … [View More]wired connection and on a VPN (we have more than one in use) different again. To help with this I have put together a script that will try pip, if it fails will check the current location for pac.pac, get and parse it for proxies, correct a common incompatibility and then try each discovered proxy and if sill none work fall back to a list of common proxies. Once it finds a working setting it advises the user what to use in the current situation. Obviously, since the script is intended to work out how to get to PyPi, I needed to stick 100% to the standard libraries. -----Original Message----- From: Paul Moore <p.f.moore(a)gmail.com> Sent: 06 April 2020 09:11 To: Stephen J. Turnbull <turnbull.stephen.fw(a)u.tsukuba.ac.jp> Cc: Pete Wicken <petewicken(a)gmail.com>; Python-Ideas <python-ideas(a)python.org> Subject: [Python-ideas] Re: macaddress or networkaddress module On Mon, 6 Apr 2020 at 04:14, Stephen J. Turnbull <turnbull.stephen.fw(a)u.tsukuba.ac.jp> wrote: > That's OK for developers on their "own" machines, but I think there > are still a lot of folks who need approval from corporate (though I > haven't heard from Paul Moore on that theme in quite a while, maybe > it's gotten better?) Howdy :-) It's actually got a bit worse, to the point where I don't bother trying to fight the system as much any more, so things have gone quieter because of that :-) But yes, I still believe that there are important use cases for code that only uses the stdlib, and "not being able to get to the internet" is a valid use case that we shouldn't just dismiss. On the other hand, with regard to the comment you were replying to: > Up to around a decade ago, installing third-party libraries > was a huge mess, but nowadays, PyPI works, Python comes with pip > pre-installed, and telling developers they need internet access > when they first start a new project isn’t considered onerous. I think all of that is true. What I think is more important, though, is around things that aren't so much "projects" as "scripts", and that *distributing* such scripts that depend on 3rd party libraries is still problematic. Here, I'm not talking about making full-fledged packages, or py2exe style fully bundled apps, but "here's this monitoring script I wrote, let's ship it across 10 production systems and why don't you try it out on your dev boxes?" There's a *significant* step change in complexity between doing that if the script depends on just the stdlib, versus if it depends on a 3rd party module. And we don't have a really good answer to that yet (zipapps, and tools like shiv that wrap zipapps, are pretty good in practice, but they are not well known or commonly used). Paul _______________________________________________ Python-ideas mailing list -- python-ideas(a)python.org To unsubscribe send an email to python-ideas-leave(a)python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/5BN5D… Code of Conduct: http://python.org/psf/codeofconduct/ [View Less]

3 3

PEP 472 -- Support for indexing with keyword arguments
by Andras Tantos Aug. 11, 2020

Aug. 11, 2020

Hello! I'm not sure I'm addressing the right audience here, so please direct me to the appropriate channel if that's the case... My name is Andras Tantos and I'm working on a Python library desscribing HW designs. I came across this problem of __getitem__ and co. not supporting kwargs. Apparently this extension was proposed and rejected as PEP 472. Apart from my use-case, which is arguably a corner-case and not worth modifying the language for, I believe there are two important use-… [View More]cases that are worth considering with the latest improvements in the language: 1. With the recent type-hint support, the feature could be made way more descriptive if this PEP got implemented. For example, instead of doing the following: def func(in: Dict[str, int]) one could write: def func(in: Dict[key=str, value=int]) 2. It would also make 'generic classes' much cleaner to implement, similar to the way type-hints look. Consider the following code: class_Generic(object): Specializations = [] @classmethod def__getitem__(cls, *args): name = f"Generic_{len(cls.Specializations)}" Specialized = type(name, (cls,), {"specials": tuple(args)}) cls.Specializations.append(Specialized) returnSpecialized def__init__(self, value= None): self.value = value def__str__(self): ifhasattr(self, "specials"): return(f"[{type(self)}- "+ ",".join(str(special) forspecial inself.specials) + f"] - {self.value}") else: return(f"[{type(self)}- GENERIC"+ f"] - {self.value}") Generic = _Generic() #g = Generic() - fails because of no specialization is given s1 = Generic[12]() s2 = Generic[42]("Hi!") print(s1) print(s2) Running this simple example results in: python3 -i python_test.py [<class '__main__.Generic_0'> - 12] - None [<class '__main__.Generic_1'> - 42] - Hi! You can see how the specialized parameters got passed as well as the ones to '__init__'. Obviously, in real code the idea would be to filter generic parameters and set up 'Specialized' with the right set of methods and arguments. Now, without kwargs support for __getitem__, it's impossible to pass named arguments to the specialization list, which greatly limits the usability of this notation. I don't know how convincing these arguments and use-cases are for you, but could you advise me about how to start the 'ball rolling' to drum-up support for re-activating this PEP? Thanks again, Andras Tantos [View Less]

24 74

Improve handling of Unicode quotes and hyphens
by Steve Barnes Aug. 6, 2020

Aug. 6, 2020

Hi All, Apologies if this has already been discussed to death. Python 3 allows Unicode characters in strings and identifiers but the actual quotation marks are only accepted in plain ASCII, i.e. the following all successfully initialise strings: ``` S1 = "Double Quoted" # Opened and closed with chr(34)0x22 S2 = 'Single Quoted' # Opened and closed with chr(39)0x27 ``` But the following all result in an error - "SyntaxError: invalid character in identifier": ``` S1 = "Double Quoted" # Opened … [View More]with \u201c and closed with \u201d S2 = 'Single Quoted' # Opened with \u2018 and closed with \u2019 ``` To the experienced eye, and depending on the character font used, it is "obvious" what the problem is! The wrong quotation marks were used. The big problem, especially for beginners, is that the same keys were typed, just in the "wrong" editor or even the wrong editor mode or context I have found that in Outlook if the font is FixSys or I am replying to a plain text email it is fine but otherwise it is "helpful" - unfortunately, especially on Windows, "wrong" editors abound and include, but are not limited to, MS-Outlook, MS-Word, some online editing environments such as Quora. On top of that is the helpful substitution of a m-hyphen for minus when you press space a word later so: A = 3 - 2 # With a space syntax error due to \u2013 A = 3 - 2 # No Space or CR after I last typed it is OK as 0x2d Use cases that catch people out: 1. Sending a code snipped by email using Outlook 2. User manuals written in MS-Word - (many peoples work environment) 3. Articles published on Quora - people expect to be able to copy and paste the code for some reason. I am sure that many us have encountered these issues or similar. What can be done? 1. Persuade Microsoft, and others, to stop being so helpful by default - good luck with that! 2. Tell all users that they need to use a "proper" editor or IDE - This seems like adding an additional barrier to new & casual users. 3. Better yet tell them to use a "proper" OS like .... - At the very least many of us have to use Windows at work. 4. Start accepting hyphens as minus & Unicode quotation marks - this would be the ideal answer for pasted code but has a lot of possible things to iron out such as do we require that the quotes match and are in the typographically correct order. It is also quite a big & complex change to the python interpreter. 5. Normalise the input to the python interpreter (at least for these characters and possibly a few others) so that entering or reading from a file S1 = "Double Quoted" becomes S1 = "Double Quoted", etc. - this should be a easier change to the interpreter but, from a purist point of view, could be said to make us as bad as the others because we are not honouring what the user entered. 6. Change the error message "SyntaxError: invalid character in identifier" to include which character and it's Unicode value so that it becomes "SyntaxError: invalid character 0x201c " in identifier" - this is almost certainly the easiest change and fits well with explicit is better than implicit but still leaves it to the user to correct the erroneous input (which could be argued is both good and bad). I would like to suggest that an incremental approach might be the best - clarifying the existing error message being the thing that should not break anything and either substituting for problem characters or processing them "properly" as a later enhancement. Steve Barnes [View Less]

16 46

python interpreter docker builder
by Antonio Cavallo July 15, 2020

July 15, 2020

Hi is there any interest (or anyone has done it before), building the python interpreter using docker? The basic idea is building the toolchain (gcc) and on top of that the python interpreter. On mac/linux it can build natively, but it can use docker to target linux from mac/windows. Thanks

5 15

Python __main__ function
by redradist＠gmail.com June 29, 2020

June 29, 2020

Hi all, In Python we often use the following syntax to call the main logic of script when it was ran: ```python def main(): pass # whatever should be done for `python ./script.py` if __name__ == '__main__': main() ``` Maybe it is a time to introduce the new module level function like __main__ ? Consider the following code: ```python def __main__(): pass # whatever should be done for `python ./script.py` ``` It is much easy an less code to write ;) Under-hood `Python` just will … [View More]

14 20

Make `del x` an expression evaluating to `x`
by Eric Wieser June 12, 2020

June 12, 2020

TL;DR: should we make `del x` an expression that returns the value of `x`. ## Motivation I noticed yesterday that `itertools.combinations` has an optimization for when the returned tuple has no remaining ref-counts, and reuses it - namely, the following code: >>> for v in itertools.combinations([1, 2, 3], 1): ... print(id(v)) ... del v # without this, the optimization can't take place 2500926199840 2500926199840 2500926199840 will print the same id … [View More]

15 50

PYTHONLOGGING env variable
by Bar Harel June 6, 2020

June 6, 2020

Another idea I've had that may be of use: PYTHONLOGGING environment variable. Setting PYTHONLOGGING to any log level or level name will initialize logging.basicConfig() with that appropriate level. Another option would be that -x dev or a different -x logging will initialize basic config. Will be useful mostly for debugging purposes instead of temporarily modifying the code. Kinda surprised it doesn't exist tbh. Bar Harel

9 22

Advanced Debugger Support C-API is incomplete?
by Andrew Pashkin June 6, 2020

June 6, 2020

I want to implement my own version of _PyThread_CurrentFrames() <https://github.com/python/cpython/blob/e838a9324c1719bb917ca81ede8d766b5cb5…>, but it seems like it's not possible using public C-API. The comment <https://github.com/python/cpython/blob/e838a9324c1719bb917ca81ede8d766b5cb5…> in the source code of _PyThread_CurrentFrames() says that lists of interpreters and threads can be mutated even when GIL is held and so HEAD_LOCK() should be used. At the same time Advanced … [View More]

1 1

Optional keyword arguments
by James Lu June 2, 2020

June 2, 2020

Many a python programmer have tired to see code written like: def bar(a1, a2, options=None): if options is None: options = {} ... # rest of function syntax if argument is not passed, evaluate {} and store to options def foo(options:={}): pass syntax if argument is not passed or is None, evaluate {} and store to options* def foo(options?={}): pass The Zen of Python states "there shouldn't be two ways to do the same thing." Thus, only one of ":=" or "?=" should be adopted. They … [View More]

23 151

How to propose a change with tests where the failing test case (current behaviour) is bad or dangerous
by Steve Barnes June 1, 2020

June 1, 2020

There is a long standing issue, (or at least I consider it a poor feature some others consider it compliance with other existing tools), with a couple of python standard libraries that I would like to raise an issue for & PR(s) to address but when it comes to putting tests in place the failing test case can potentially crash the host machine, sometimes in an unrecoverable manner, due to the potential to generate an infinite size file. Is there a way of proposing a change with tests that … [View More]

9 21

Python GIL Thoughts
by redradist＠gmail.com June 1, 2020

June 1, 2020

Hi all, I am very exciting about the sub-interpreters ... but I do not like some parts ... Consider the following code: ```python import _xxsubinterpreters as interpreters import threading import textwrap as tw import marshal if __name__ == '__main__': # Create a sub-interpreter interpid = interpreters.create() # If you had a function that generated some data arry = list(range(0,100)) # Create a channel channel_id = interpreters.channel_create() # Pre-populate … [View More]the interpreter with a module interpreters.run_string(interpid, "import marshal; import _xxsubinterpreters as interpreters") # Define a def run(interpid, channel_id): # !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Still is run in thread which is use GIL !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! interpreters.run_string(interpid, tw.dedent(""" arry_raw = interpreters.channel_recv(channel_id) arry = marshal.loads(arry_raw) result = [1,2,3,4,5] # where you would do some calculating result_raw = marshal.dumps(result) interpreters.channel_send(channel_id, result_raw) """), shared=dict( channel_id=channel_id ), ) inp = marshal.dumps(arry) interpreters.channel_send(channel_id, inp) # Run inside a thread t = threading.Thread(target=run, args=(interpid, channel_id)) t.start() # Sub interpreter will process. Feel free to do anything else now. output = interpreters.channel_recv(channel_id) interpreters.channel_release(channel_id) output_arry = marshal.loads(output) print(output_arry) ``` Add some `async` execution of `interpreters.run_string_async` and other async functions !! Also regarding the threads ... I think it is better to make `threading.Thread` as green `Thread` !! I know, I know, `Thread` is not possible due to `sync io` ... But we can to change underling `api` from `sync io` to `async io` !! User will notice nothing, but performance will be increased mush more !! Consider the following execution of `threading.Thread` in Python: ``` gil_lock.acquire() for green_thread in green_threads: thread.execute() gil_lock.release() ``` If we implement in such way we will be able to reduce resource used in system, but performance would be the same !! Sub-Interpreter would be the something like Workers in JavaScript, but much more flexible !! What we will have at the end: 1) `threading.Thread` is a green thread that do not consume a lot of resources but still useful !! 2) Real parallelizm is done within sub-interpreters like workers in JavaScript, but much more flexible Say what do you think ? [View Less]

9 22

datetime.time input restrictions
by Ben Axelrod May 29, 2020

May 29, 2020

I was wondering why have restrictions on the datetime.time constructor arguments? For example, why can't you enter 1.5 minutes or 90 seconds to create a datetime object with 1 minute and 30 seconds? >>> datetime.time(minute=1.5) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: integer argument expected, got float >>> datetime.time(second=90) Traceback (most recent call last): File "<stdin>", line 1, in <module> … [View More]

2 1

Expose PyFloat_AsDouble to Python
by Mark Dickinson May 29, 2020

May 29, 2020

Here's the short version: * "Convert float-like object to a float" is a useful operation * For a C extension, there's an obvious way to spell that operation: use `PyFloat_AsDouble` * In pure Python, there's no Obvious Way To Do It * Proposal: expose the equivalent of `PyFloat_FromDouble(PyFloat_AsDouble(obj))` to Python * Question: where should it go? Longer version: Various parts of CPython that expect a float will accept anything "float-like"; examples include most of the `math` module … [View More]functions, the `struct` module, `ctypes`, `statistics`, old-style string formatting with `"%f"` and the like, anything that uses `PyArg_ParseTuple` with the `'d'` format converter, anything that uses argument clinic and declares an input as `double`, and more. This is a Good Thing: it means that as a Python user, you can make your own types compatible with these operations simply by giving them a `__float__` method. Now suppose that as a Python user and 3rd party library writer, I want to write my own function that similarly accepts anything float-like, following the same rules that core Python uses. If I'm writing a C extension module, this is trivially easy: I use `PyFloat_FromDouble`, which is part of the stable ABI. However, if I'm writing in pure Python, it's surprisingly awkward to clearly express the equivalent behaviour, and I'd like to have an obvious way to do it. Some of the non-obvious ways: 1. Use the `float` constructor. This _ought_ to be the one obvious way to do it: it'll certainly accept anything float-like and convert it to a float. But it will also accept instances of `str`, `bytes`, `bytearray`, and anything supporting the buffer protocol, so if you want to exclude those (for example, because passing a non-number to your numeric code is likely to be a coding error, and you'd like to catch it as such) you need to do a LBYL check. 2. Call the object's `__float__` method. But this is fraught with peril, too: for a proper equivalent, you need to be careful to look up `__float__` on the type, not the object itself. And then a new version of Python changes `PyFloat_AsDouble` to also accept objects with `__index__`, and suddenly your version no longer matches what Python does. (This happened.) 3. Leverage the `math` module's existing ability to do this. For example, I can do `x_as_float = math.copysign(x, x)`. This works, giving me _exactly_ the semantics that I want. But I think it would be stretch to call this an *obvious* way to do it. 4. Write your own small C extension containing a single function that looks like `return PyFloat_FromDouble(PyFloat_AsDouble(obj))` (with extra error checking, fast path for exact floats, etc.). So my modest proposal is: expose the conversion represented by `PyFloat_AsDouble` to Python somewhere: either on the `float` type itself, or somewhere in the standard library. My question for everyone on this list is: _if_ it were to be added, where should it go? (There's also the "what should it be called" question, of course.) I have a proof-of-concept PR [1] that exposes this in the `operator` module as `operator.as_float`. See also the discussion on the tracker [2]. [1] https://github.com/python/cpython/pull/20481 [2] https://bugs.python.org/issue40801 [View Less]

3 3

An HTTP API to list versions of Python
by Julien Palard May 29, 2020

May 29, 2020

We do not have a standard way to get (programatically) informations about Python versions, this leads to having to commit to numerous places during each releases, and some other places never get the information (like old docs, see [1]). Maybe we should provide an HTTP API listing Python versions, like: { "current_stable": "3.8.3", "versions": [ { "branch": "master", "status": "dev", "version": "3.10", "release_manager":… [View More] "Łukasz Langa", "doc_languages": ["en", "es", "fr", "ja", "ko", "pt-br", "zh-cn"] }, { "branch": "3.9", "latest": "3.9.0b1", "status": "features", "pep": 596, "version": "3.9", "release_manager": "Łukasz Langa", "doc_languages": ["en", "fr", "ja", "ko", "pt-br", "zh-cn"] }, { "branch": "3.8", "latest": "3.8.3", "status": "bugfix", "pep": 569, "version": "3.8", "release_manager": "Łukasz Langa", "first_release": "2019-10-14", "end_life": "2024-10", "doc_languages": ["en", "fr", "ja", "ko", "pt-br", "zh-cn"] }, { "branch": "3.7", "latest": "3.7.7", "status": "bugfix", "pep": 537, "version": "3.8", "release_manager": "Ned Deily", "first_release": "2018-06-27", "end_life": "2023-06-27", "doc_languages": ["en", "fr", "ja", "ko", "pt-br"] } ... (Don't really mind this particular example, it's a random first though, the doc_languages should probably be a dict instead, with links to the actual docs, ...) It could be used to (and maybe avoid a commit) to: - build the documentation version switcher and languages switchers - toggle the red deprecated banner (to be seen on [2] if you missed it) - generate the versions table at https://devguide.python.org/ - generate the Docs by version menu - generate the Download menu and download page of python.org - list the documentations versions to be built by docsbuild-script - probably many scripts could use it too, I can think of [3] - ... It could become more detailed about each minor versions, git tag, links to changelogs, links to the repositories, to the docs, download links, and so on. Maybe this idea will just add a commit to do between the numerous commits to already do during a release... (à la https://xkcd.com/927/), and a SPOF. What do you think? [1]: https://mail.python.org/archives/list/docs@python.org/thread/L724YZXXHQZQP4… [2]: https://docs.python.org/2.7/ [3]: https://gitlab.com/python-devs/ci-images/-/blob/master/Dockerfile#L19 -- [Julien Palard](https://mdk.fr) [View Less]

8 9

Re: Python __main__ function
by Jose Veiga May 28, 2020

May 28, 2020

While is possible to use `if __name__ == '__main__':` several times in the same script, your proposed magic function `def __main__()` cannot be redefined. Not to speak about lexical scope differences between one approach and the other. So your proposal is reducing features instead of expanding them. Jose Veiga

6 6

Re: Python __main__ function
by Chris Angelico May 28, 2020

May 28, 2020

On Fri, May 29, 2020 at 5:54 AM <tritium-list(a)sdamon.com> wrote: > > > -----Original Message----- > > From: Chris Angelico <rosuav(a)gmail.com> > > > > People can already put all their main logic into a function. If you want > to unit- > > test your main function, that's the best way to do it. > > The trouble is, how much goes into main() and how much into if __name__ > > == '__main__'? For example: Does your main function accept a … [View More]

1 0

Allowing assertEqual etc to take multiple items
by Ram Rachum May 28, 2020

May 28, 2020

Hi guys! I'm a unittest newbie, so it's possible there's a better solution than what I'm proposing here. If there is, please let me know. I recently submitted this PR <https://github.com/more-itertools/more-itertools/pull/430/files>, and I had to use assertSequenceEqual twice, just because I wanted to compare 3 sequences to each other rather than 2. Wouldn't it be possible to make assertEqual and its entire family (including assertSequenceEqual and the dozen of other methods) accept an … [View More]

4 4

Re: Optional keyword arguments
by Chris Angelico May 27, 2020

May 27, 2020

On Thu, May 28, 2020 at 10:42 AM Rob Cliffe <rob.cliffe(a)btinternet.com> wrote: > > > > On 26/05/2020 20:03, Chris Angelico wrote: > > On Wed, May 27, 2020 at 2:51 AM Steven D'Aprano <steve(a)pearwood.info> wrote: > >> But doing otherwise, having Undef be *not an object* but a kinda ghost > >> in the interpreter, is a huge language change and I doubt it would be > >> worth it. > >> > > But is it a huge change? I thought so … [View More]

1 0

len(path) == len(str(path))
by Ram Rachum May 25, 2020

May 25, 2020

Hi everyone, I submitted a PR today, and Serhiy decided it needs a discussion on python-ideas and agreement from core developers before it could go forward. BPO: https://bugs.python.org/issue40752 PR: https://github.com/python/cpython/pull/20348 Today I wrote a script and did this: sorted(paths, key=lambda path: len(str(path)), reverse=True) But it would have been nicer if I could do this: sorted(paths, key=len, reverse=True) So I implemented `PurePath.__len__` as `str(len(path))`… [View More]

23 46

type hints : I'd like to suggest allowing unicode → as an alternative to ->
by Thierry Parmentelat May 24, 2020

May 24, 2020

well it’s all in the title the specific character that I am referring to is this one In [1]: print("\u2192”) → https://unicode-table.com/en/2192/ —— just curious about how people would feel about taking better advantage of non-ascii characters when that seems to make sense fyi here’s how both options appear in a markdown-based website thanks !

24 68

Python tagged unions based on dataclasses
by Andrey Cizov May 23, 2020

May 23, 2020

I have developed a library to introduce tagged unions to python that uses dataclasses to define disjoint members of the tagged union (by defining them as Optional fields). With some additional validation I make sure that only one of the fields is not None. I find that it also fits well with the existing analysis tools and IDEs (e.g. PyCharm) and doesn’t require any additional work in order to be supported. I would like to see some criticism and whether that could potentially be a good … [View More]

3 5

Enhance list.index with a comparison function
by Steven D'Aprano May 23, 2020

May 23, 2020

Have a look at this short thread on discuss: https://discuss.python.org/t/searching-array-of-data-objects/4251/1 After I answered that question, it dawned on me that I have probably written something like that loop, or variations of it, a thousand times: for obj in somelist: if comparison(obj, needle): do_something(obj) break and now it's a thousand and one times *wink* So here's a thought... suppose we gave list.index an optional keyword- only … [View More]

7 11

Re: [Python-Dev] Re: PEP 618: Add Optional Length-Checking To zip
by Steven D'Aprano May 21, 2020

May 21, 2020

On Fri, May 15, 2020 at 05:46:43PM +0200, Antoine Pitrou wrote: > On Fri, 15 May 2020 10:46:25 -0400 > David Mertz <mertz(a)gnosis.cx> wrote: > > > > > 1. +1 itertools.zip_strict function > > > 2. +1 zip.strict(*args) > > > 3. +1 zip(*args, mode='strict') # mode='shortest' by default > > > 4. +0 zip(*args, strict=True) > > > > > > > Mostly I agree with Steven on relative preference: > > > > itertools.… [View More]zip_strict() +1 > > zip.strict() +0.5 > > zip(mode='strict') +0 > > zip(strict=True) -0.5 > > For me: > > * zip(strict=True) +1 > * zip(mode='strict') -0 > * itertools.zip_strict() -0.5 > * zip.strict() -1 (but really, I'd like to make this -1e10) I spent a significant amount of time and mental energy explaining in detail why a boolean flag is a poor API and is objectively the wrong interface here. This is not just a matter of personal taste: there are reasons why a flag is wrong here. Flags are for combining independent and orthogonal settings which can be combined. This is not such a feature. I think it is objectively the worst API on the table here, for reasons already discussed. - Your preferred option makes the strict zip version a second-class citizen of the language; - your preferred option is the least open to future enhancements; - your most hated option is the one which follows the Zen of Python the most closely (namely, the koan about having more namespaces); - and is the most object-oriented solution (it's effectively a method); - and most importantly you explicitly *oppose* every alternative API, giving them negative preferences; you would rather not have this zip variant at all than have an interface other than a boolean flag. If I needed this function[1], I'd accept it even if it were spelled `xyghasx.peyahc.flihaj()` and required me to set a global variable to make it work. So I can't help but interpret your total opposition to every other interface as a strong sign that you don't really need this zip variant at all. If your opinion is typical, perhaps we should just reject the PEP. Could you explain why you believe a bool flag is the only suitable interface? Objective reasons preferred please. The same goes for everyone else who gave a vote. If you have an objective reason for wanting this strict zip function to be a second class citizen that cannot be easily extended in the future, then please explain why. [1] I don't think I do. If I had my preference, thinking only of my own needs, I'd probably reject the PEP. But I acknowledge that others seem to need it. -- Steven [View Less]

10 20

PEP 618: Add Optional Length-Checking To zip
by Brandt Bucher May 20, 2020

May 20, 2020

I have pushed a first draft of PEP 618: https://www.python.org/dev/peps/pep-0618 Please let me know what you think – I'd love to hear any *new* feedback that hasn't yet been addressed in the PEP! Brandt

26 72

str.strip: argument maxstrip
by computermaster360 . May 20, 2020

May 20, 2020

I often find myself in a need of stripping only a specified amount of characters from a string (most commonly a single character). I have been implementing this in an ad-hoc manner, which is quite inelegant: def rstrip1(txt, chars): if txt is None: return None elif any(txt.endswith(c) for c in chars): return txt[:-1] else: return txt I would appreciate if str.split, str.lstrip, str.rsplit functions had a `maxstrip` argument, similar to the `maxsplit` argument of `str.split`, … [View More]

7 17

Ensuring asserts are always on in a file
by Alex Hall May 19, 2020

May 19, 2020

Some people (like myself, or the coworkers of [this person](https://mail.python.org/archives/list/python-ideas@python.org/threa…) just like to use asserts as a convenient way to check things. We don't want asserts to ever be turned off. Maybe there could be some kind of compiler directive which means "in this file, even with -O, keep the asserts". Maybe the line `assert __debug__`?

5 7

Formally deprecate urllib.request.urlretrieve() and urllib.request.urlcleanup()
by remi.lapeyre＠henki.fr May 18, 2020

May 18, 2020

Both urlretrieve() and urlcleanup() were kept for backward compatibility with Python2 but they were never deprecated like urllib.request.URLOpener and urllib.request.FancyURLOpener. Now that Python2 has been definitely sunset, can we raise a deprecation warning so that they can be removed in the future?

2 4

islice with actual slices
by Ram Rachum May 17, 2020

May 17, 2020

Here's an idea I've had. How about instead of this: itertools.islice(iterable, 7, 20) We'll just have: itertools.islice(iterable)[7:20] Advantages: 1. More familiar slicing syntax. 2. No need to awkwardly use None when you're interested in just specifying the end of the slice without specifying the start, i.e. islic(x)[:10] instead of islice(x, None, 10) 3. Doesn't require breaking backwards compatibility. What do you think?

13 79

Re: Auto-assign attributes from __init__ arguments
by Alex Hall May 17, 2020

May 17, 2020

Steven D'Aprano wrote: > Proposal: > We should have a mechanism that collects the current function or > method's parameters into a dict, similar to the way locals() returns all > local variables. > This mechanism could be a new function,or it could even be a magic local > variable inside each function, similar to what is done to make super() > work. But for the sake of this discussion, I'll assume it is a function, > parameters(), without worrying about whether it is a … [View More]

1 0

Auto-assign attributes from __init__ arguments
by Lewis Ball May 16, 2020

May 16, 2020

Hi All, First of all, if this is something which has been discussed in the past the please point me in the right direction. *Problem:* When creating classes in Python, I find myself writing the __init__ method in a very similar way a lot of the time, that is: ``` def __init__(self, argument_1, argument_2, argument_3=None): self.argument_1 = argument_1 self.argument_2 = argument_2 self.argument_3 = argument_3 # then maybe some other attribute setting and logic follows ``` … [View More]Every argument of __init__ gets a corresponding attribute with the same name. This means that each `argument_i` has been typed 3 times, which seems overly-verbose as well as being easy to mistype. This pattern is easy to find in various popular python libraries, and in some it is actually enforced. For example, I do quite a bit of work with classifiers using the sklearn estimator API, and for various reasons sklearn enforce this pattern for an __init__ (see here <https://scikit-learn.org/stable/developers/develop.html#instantiation> if interested). Here is an example of this pattern from the standard library (from textwrap.TextWrapper): ``` def __init__(self, width=70, initial_indent="", subsequent_indent="", expand_tabs=True, replace_whitespace=True, fix_sentence_endings=False, break_long_words=True, drop_whitespace=True, break_on_hyphens=True, tabsize=8, *, max_lines=None, placeholder=' [...]'): self.width = width self.initial_indent = initial_indent self.subsequent_indent = subsequent_indent self.expand_tabs = expand_tabs self.replace_whitespace = replace_whitespace self.fix_sentence_endings = fix_sentence_endings self.break_long_words = break_long_words self.drop_whitespace = drop_whitespace self.break_on_hyphens = break_on_hyphens self.tabsize = tabsize self.max_lines = max_lines self.placeholder = placeholder ``` With a quick scan of the top 50 or so most used python packages, *1 in 4* __init__ methods that takes arguments has the line `self.argument_i = argument_i` for every single argument, with several of them having 10+ arguments. *Suggestion:* A new built-in called something like `assign()` which would assign every single __init__ arg to a corresponding attribute. e.g. the snippet from above could be rewritten to: ``` def __init__(self, argument_1, argument_2, argument_3=None): assign() # other init logic goes here ``` This could alternatively be implemented as a decorator, like so ``` @assign def __init__(self, argument_1, argument_2, argument_3=None): # other init logic goes here ``` but then this requires a `pass` if no other logic is needed inside the __init__. There may also be some other syntax for this which would be even easier to use. Is this something that others would find useful? Thanks, Lewis [View Less]

15 34

Resetting peak memory metric in tracemalloc without touching other traces
by wilson.huon＠gmail.com May 14, 2020

May 14, 2020

Hi, It would be helpful for us if tracemalloc had a function that reset the peak memory usage counter, without clearing the current traces. At the moment, I don't think there's a way to find the peak memory of a subset of the code since the initial tracemalloc.start() call, without calling tracemalloc.clear_traces(). The latter disturbs other parts of the tracing. Specifically, it might be a function like (pseudo-implementation): def reset_peak_memory(): # in _tracemalloc.c … [View More] tracemalloc_peak_trace_memory = tracemalloc_traced_memory; This would allow easily determining the peak memory usage of a specific piece of code, without disturbing all of the traces. For example, the following would set specific_peak to the highest size of traced memory of just line X: tracemalloc.start() # ... code where allocations matter, but the peak does not ... peak_memory_doesnt_matter() tracemalloc.reset_peak_memory() peak_memory_is_important() # X _, specific_peak = tracemalloc.get_traced_memory() # ... more code with relevant allocations ... peak_memory_doesnt_matter() tracemalloc.stop() As sketched above, the implementation of this should be quite small, with the core being the line mentioned above, plus all the required extras (locking, wrapping, documentation, tests, ...). Thoughts? Full motivation for why we want to do this: In <https://github.com/stellargraph/stellargraph>, we're using the tracemalloc module to understand the memory usage of our core StellarGraph graph class (a nodes-and-edges graph, not a plot, to be clear). It stores some NumPy arrays of feature vectors associated with each node in the graph, along with all of the edge information. Any of these pieces can be large, and we want to keep the resource usage as small as possible. We're monitoring this by instrumenting the construction: start from a raw set of nodes (including potentially large amounts of features) and edges, and build a StellarGraph object, recording some metrics: 1. the time 2. the total memory usage of the graph instance 3. the additional memory usage, that's not shared with the raw data (in particular, if the raw data is 1GB, it's useful to know whether a 1.5GB graph instance consists of 0.5GB of new memory, or 1.5GB of new memory) 4. the peak memory usage during construction 2, 3 and 4 we record using a combination of tracemalloc.take_snapshot() and tracemalloc.get_traced_memory(), something like: def diff(after, before): return sum(elem.size_diff for after.compare_to(before, "lineno")) snap_start = take_snapshot() raw = load_data_from_disk() snap_raw = take_snapshot() # X graph = create_graph(raw) snap_raw_graph = take_snapshot() _, mem_peak = get_traced_memory() # 4 del raw snap_graph = take_snapshot() mem_raw = diff(snap_raw, snap_start) # baseline mem_graph = diff(snap_graph, snap_start) # 2 mem_graph_not_shared = diff(snap_raw_graph, snap_raw) # 3 ('measure_memory' in <https://nbviewer.jupyter.org/github/stellargraph/stellargraph/blob/93fce461…> has all the gory details.) Unfortunately, we want to ignore any peak during data loading: the peak during create_graph is all we care about, even if the overall peak (in data loading) is higher. That is, we want to only consider the peak memory usage after line X. One way to do this would be to call clear_traces() at X, but this invalidates the traces used for the 2 and 3 comparisons. I believe tracemalloc.reset_peak_memory() is the necessary function to call at X. (Why do we want to ignore the peak during data loading? The loading is under the control of a user (of stellargraph) as it's typically done via Pandas or NumPy and those libraries are out of our control and offer a variety of options for tweaking data-loading behavior, whereas the internals of the `StellarGraph` instance are in our control and not as configurable by users.) Thanks, Huon Wilson [View Less]

3 2

Making asserts non-recoverable.
by chris.the.developer.0＠gmail.com May 14, 2020

May 14, 2020

How adverse would you guys feel about a change to the way asserts are handled so that they are not recoverable? Asserts could latch on to the first failed assert and always throw an AssertionError on subsequent assert statements. Another way would be for subsequent asserts to turn off after the first failure. As long as developers keep the contract that asserts should never fail, they wont notice this change; and it blocks developers from writing code that is dependent on asserts.

10 13

Sanitize filename (path part) 2nd try
by Steve Jorgensen May 13, 2020

May 13, 2020

Based on responses to my previous proposal, I am convinced that it was over-ambitious and not appropriate for inclusion in the Python standard library, so starting over with a more narrowly scoped suggestion. Proposal: Add a new function (possibly `os.path.sanitizepart`) to sanitize a value for use as a single component of a path. In the default case, the value must also not be a reference to the current or parent directory ("." or "..") and must not contain control characters. When an … [View More]

9 17

Sanitize filename (path part)
by Steve Jorgensen May 12, 2020

May 12, 2020

I believe the Python standard library should include a means of sanitizing a filesystem entry, and this should not be something requiring a 3rd party package. One of reasons I think this should be in the standard lib is because that provides a common, simple means for code reviewers and static analysis services such as Veracode to recognize that a value is sanitized in an accepted manner. What I am envisioning is a function (presumably in `os.path` with a signature roughly like {{{ … [View More]

7 19

Equality between some of the indexed collections
by Ahmed Amr May 9, 2020

May 9, 2020

I'd like to take your opinion on modifying some of the indexed collections like tuples, lists, arrays to evaluate its equality to True when having the same items in the same indexes. Currently, when comparing a list of items to an array of the same items for equality (==) it returns False, I'm thinking that it would make sense to return True in that context, as we're comparing item values and we have the same way of indexing both collections, so we can compare item values. So what do you … [View More]

24 104

Introduce 100 more built-in exceptions
by Ram Rachum May 9, 2020

May 9, 2020

Hi, Here's something I wanted in Python for many years. If this has been discussed in the past, please refer me to that discussion. On one hand, it's something that I can't imagine the python-dev community supporting. On the other hand, it would maintain backward compatibility. I wish there were a 100 more built-in exceptions in Python, that will be very specific about what went wrong. If I do this: >>> x, y = range(3) I know it'll raise a ValueError, because I've memorized … [View More]

16 48

General methods
by Serhiy Storchaka May 8, 2020

May 8, 2020

Methods in Python are defined as functions in the class namespace. When call the method of the object, the function will be called with the object as the first argument. And furthermore, unbound methods can be called with passing self as the first argument. For example, str.upper('abc') returns 'ABC'. So class can be considered as a namespace for functions related to objects of the specified type. For methods defined in Python you can pass arbitrary object as self. But in methods defined … [View More]in C it should be an instance of the class in which the method was defined. On one hand, it is very convenient -- you cab be sure that self is binary compatible with the specified class. On other hand, it restricts you. I propose to add the METH_GENERAL flag, which is applicable to methods as METH_CLASS and METH_STATIC (and is mutually incompatible with them). If it is set, the check for the type of self will be omitted, and you can pass an arbitrary object as the first argument of the unbound method. I have several use cases for this. 1. Bytes and bytearray methods. Bytes and bytearray has a lot of of sequence-like and string-like methods. They also implement the buffer protocol. There are other objects which implement the buffer protocol: memoryview, BytesIO, array.array, mmap.mmap, ctypes arrays, NumPy arrays, but they lack most of these methods. To use these methods (for example index()) you need to copy the content to a bytes or bytearray, that invalidates the purpose of the buffer protocol which was designed to avoid copying of binary data. With general methods we can make bytes.index() be applicable to any object which supports the buffer protocol. bytes.index() and bytearray.index() will be equivalent, but the difference between bytes.split() and bytearray.split() will be in the result type. 2. Set methods. Some set methods accept arbitrary number of arguments and accept not only sets, but any iterables. Sou you can get a union of two lists for example: >>> set().union([1, 2], [2, 3]) {1, 2, 3} It does work because a union with an empty set is a no-op. This trick does not work with other methods. You have to convert the first iterable to set explicitly. >>> set([1, 2]).symmetric_difference([2, 3]) {1, 3} With general methods we can make unbound set methods accepting arbitrary iterables and convert the first one to set implicitly (or avoid creating a set if it is possible). We could use set.symmetric_difference([1, 2], [2, 3]). Maybe there are other use cases. But I do not suggest making all methods of builtin types general: only if there is a more general protocol (as the buffer protocol or the iterator protocol) and there is a profit of using such protocol without creating an object of the corresponding type explicitly. For example, I think that str methods should not be general, despite the fact that any object can be converted to string. Implicit conversions to str does not have profit and may hide bugs. Other example -- float.as_integer_ratio() should not accept int, despite the fact that most function which accept float implicitly convert int to float. It is a lossy conversion for large integers, and the loss will affect the result. I used the term "general methods" to name this feature, but if there are better proposition I will use it. [View Less]

5 4

Adding a "once" function to functools
by Tom Forbes May 7, 2020

May 7, 2020

Hello, I would like to suggest adding a simple “once” method to functools. As the name suggests, this would be a decorator that would call the decorated function, cache the result and return it with subsequent calls. My rationale for suggesting this addition is twofold: First: It’s fairly common to use `lru_cache()` to implement this behaviour. We use this inside Django (example <https://github.com/django/django/blob/77aa74cb70dd85497dbade6bc0f394aa41e88…>), internally in other projects … [View More]at my workplace, inside the stdlib itself <https://github.com/python/cpython/blob/2fa67df605e4b0803e7e3aac0b85d851b4b4…> and in numerous other projects. In the first few pages of a Github code search <https://github.com/search?l=Python&q=%22functools.lru_cache%22&type=Code> it is fairly easy to find examples, any decorated method with no parameters is using `lru_cache()` like `once()`. Using lru_cache like this works but it’s not as efficient as it could be - in every case you’re adding lru_cache overhead despite not requiring it. Second: Implementing this in Python, in my opinion, crosses the line of “annoying and non-trivial enough to not want to repeatedly do it”. While a naive (untested) implementation might be: def once(func): sentinel = object() # in case the wrapped method returns None obj = sentinel @functools.wraps(func) def inner(): nonlocal obj, sentinel if obj is sentinel: obj = func() return obj return inner While to the people who are likely going to be reading this mailing this the code above is understandable and potentially even somewhat simple. However to a lot of people who might not have had experience with writing decorators or understand sentinel objects and their use the above code might be incomprehensible. A much more common, and in my opinion worse, implementation that I’ve seen is something along the lines of this: _value = None def get_value(): nonlocal _value if _value is None: _value = some_function() return _value Which is not ideal for obvious reasons. And these are not even including a potentially key feature: locking the wrapped function so that it is only called once if it is invoked from multiple threads at once. So, I’d like to propose adding a `once()` decorator to functools that: 1. Has a C implementation, keeping the speed on-par with `lru_cache()` 2. Ensures that the wrapped function is only called once when invoked by multiple threads For some related discussion about this idea and lru_cache, please see my thread on <https://discuss.python.org/t/reduce-the-overhead-of-functools-lru-cache-for…>discuss.python.org <http://discuss.python.org/>. [View Less]

22 48

PEP 597 -- Soft deprecation of omitting encoding
by Inada Naoki May 3, 2020

May 3, 2020

The PEP is here: https://www.python.org/dev/peps/pep-0597/ I created a thread on discuss.python.org <https://discuss.python.org/t/pep-597-raise-a-warning-when-encoding-is-omitt…> but I want more feedback. Please reply on this mail or discuss.python.org. ## Summary This PEP doesn't propose to change the default text encoding yet, but introduce warning for code using default text encoding only in dev mode. This opt-in warning is useful for: * Estimate the impact of changing the default … [View More]

1 0

Re: Adding a "once" function to functools
by Tom Forbes May 2, 2020

May 2, 2020

> Thread 2 wakes up with the lock, calls the function, fills the cache, and releases the lock. What exactly would the issue be with this: ``` import functools from threading import Lock def once(func): sentinel = object() cache = sentinel lock = Lock() @functools.wraps(func) def _wrapper(): nonlocal cache, lock, sentinel if cache is sentinel: with lock: if cache is sentinel: cache = func() return … [View More]cache return _wrapper ``` It ensures it’s invoked once, it’s fast and it doesn’t require acquiring a lock needlessly. Always wrapping a lock around a static value to account for a small edge case seems redundant, more than “it’s usually not optimal”. It’s never optimal: in multi-threaded cases you’re causing needless contention, and in single threaded cases all your calls are now slower. Seems generally more correct, even in single threaded cases, to pay the overhead only in the first call if you want `call_once` semantics. Which is why you would be using `call_once` in the first place? > On 29 Apr 2020, at 18:51, Andrew Barnert <abarnert(a)yahoo.com> wrote: > > On Apr 29, 2020, at 00:34, Tom Forbes <tom(a)tomforb.es> wrote: >> >> It’s not quite that easy, either you needlessly lock all calls or you end up invoking it twice. >> >> What you want is to acquire a lock if the cache is empty, check if another thread has filled the cache while you where waiting on the lock, call the function, fill the cache and return. > > Thread 1 sees the cache is empty, so it acquires the lock, runs the function. > > Thread 2 sees the cache is empty, so it tries to acquire the cache and blocks. > > Thread 1 finishes the function, fills the cache, and releases the lock. > > Thread 2 wakes up with the lock, calls the function, fills the cache, and releases the lock. > > You’ve now run the function twice instead of once, and the first caller got a different value than the one every subsequent caller can get. > > There are actually plenty of cases where this is acceptable (the function is safe, and idempotent, and while it’s slow it’s not so slow that occasionally running it twice is worse than locking a zillion times)—but in those cases, just not using a lock is even better. In fact, if you can compare-and-swap at the end (which requires a lock to simulate in Python, but it’s a much more granular one than locking over the whole expensive function call), you can even use it in more cases (in particular, where the function isn’t idempotent but you need to value to be). > > This is exactly why users should be encouraged to just lock the whole thing. It’s always correct. It’s usually not optimal, but often good enough. The one really common and simple case where it’s not good enough is single-threaded code. Anything else where you need to optimize, you really do need to understand the pattern you’re trying to optimize for (and the environment(s) you’re running under) and write your own function. > > [View Less]

4 7

is a
by gbs＠me.com May 1, 2020

May 1, 2020

In cases where it makes sense to do explicit type checking (with ABCs or whatever), I really detest the look of the isinstance() function. if isinstance(thing, Fruit) and not isinstance(thing, Apple): Yucky. What I really want to write is: if thing is a Fruit and thing is not an Apple: and after thinking about it on and off for a while I wonder if it might indeed be possible to teach the parser to handle that in a way that eliminates almost all possible ambiguity with the regular "is", … [View More]

7 9

Re: Function to avoid a global variable
by Chris Angelico May 1, 2020

May 1, 2020

On Sat, May 2, 2020 at 10:17 AM Terry Reedy <tjreedy(a)udel.edu> wrote: > > On 5/1/2020 7:30 PM, Chris Angelico wrote: > > > @static(called=0) > > def other_function(): > > me.called += 1 > > ... > > > > Obviously the name "me" can't be used, as it'd break a bunch of code, > > but conceptually this would be incredibly helpful. It'd also be a > > reliable idiom for recursion optimization - any "me()" is guaranteed > > … [View More]to be recursion and may potentially give info to an optimizer. > > > > Perhaps, if Python had a way to identify the current function, it > > would feel less odd to attach attributes to it. > > As one might imagine, something so 'obvious' has been proposed multiple > times. Reasons for repeated rejection. > > 1. Function bodies are compiled to code objects. There is no function > to 'bind' to and there might never be. (compile(code_body, ...)). True, but that's a bit of an edge case. I'm sure there are other edge cases too - what happens with a 'nonlocal' if it's in a code object outside of a function? (Or in the CPython implementation, what happens with the corresponding cell lookup?) If <me> causes a RuntimeError if used in such a context, or if the compilation of <me> creates some sort of reference that simply doesn't work outside of a function, it'd be fine. > 2. Python does name resolution on function bodies when called. If early > binding of <me> were possible, it would be an exception. It would have to be a keyword of a sort. Not necessarily an actual keyword, although that would definitely work. > 3. Recursion without names is really hard; hence the special > head-spinning combinator to do that. And that's a good reason to have compiler support for it. It would make true recursion simple. (Obviously mutual recursion would still require names.) > 4. The post-compile function object address would only be useful for > CPYthon, not for implementations that move objects around. Why? It's an object. It should be possible to reference it. > 5. The automatic conversion of recursion to a while loop only works for > linear tail recursion. In Python, such cases are usually better written > directly as a for-loop. True, and that would probably only be a small advantage anyway. But it's also a clarity thing, and it means you don't have to name your function inside the function and then remember to rename it. There is another objection though: 6. Decorated functions would behave differently based on whether you call by name or call by <me>. I don't have an answer to that, other than that giving programmers the option isn't a bad thing. People would just have to be aware of the distinction and know which one they want. ChrisA [View Less]

1 0