Mailman 3 May 2020 - Python-ideas

Python GIL Thoughts
by redradist＠gmail.com June 2, 2020

June 2, 2020

Hi all, I am very exciting about the sub-interpreters ... but I do not like some parts ... Consider the following code: ```python import _xxsubinterpreters as interpreters import threading import textwrap as tw import marshal if __name__ == '__main__': # Create a sub-interpreter interpid = interpreters.create() # If you had a function that generated some data arry = list(range(0,100)) # Create a channel channel_id = interpreters.channel_create() # Pre-populate the interpreter with a module interpreters.run_string(interpid, "import marshal; import _xxsubinterpreters as interpreters") # Define a def run(interpid, channel_id): # !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Still is run in thread which is use GIL !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! interpreters.run_string(interpid, tw.dedent(""" arry_raw = interpreters.channel_recv(channel_id) arry = marshal.loads(arry_raw) result = [1,2,3,4,5] # where you would do some calculating result_raw = marshal.dumps(result) interpreters.channel_send(channel_id, result_raw) """), shared=dict( channel_id=channel_id ), ) inp = marshal.dumps(arry) interpreters.channel_send(channel_id, inp) # Run inside a thread t = threading.Thread(target=run, args=(interpid, channel_id)) t.start() # Sub interpreter will process. Feel free to do anything else now. output = interpreters.channel_recv(channel_id) interpreters.channel_release(channel_id) output_arry = marshal.loads(output) print(output_arry) ``` Add some `async` execution of `interpreters.run_string_async` and other async functions !! Also regarding the threads ... I think it is better to make `threading.Thread` as green `Thread` !! I know, I know, `Thread` is not possible due to `sync io` ... But we can to change underling `api` from `sync io` to `async io` !! User will notice nothing, but performance will be increased mush more !! Consider the following execution of `threading.Thread` in Python: ``` gil_lock.acquire() for green_thread in green_threads: thread.execute() gil_lock.release() ``` If we implement in such way we will be able to reduce resource used in system, but performance would be the same !! Sub-Interpreter would be the something like Workers in JavaScript, but much more flexible !! What we will have at the end: 1) `threading.Thread` is a green thread that do not consume a lot of resources but still useful !! 2) Real parallelizm is done within sub-interpreters like workers in JavaScript, but much more flexible Say what do you think ?

9 22

datetime.time input restrictions
by Ben Axelrod May 29, 2020

May 29, 2020

I was wondering why have restrictions on the datetime.time constructor arguments? For example, why can't you enter 1.5 minutes or 90 seconds to create a datetime object with 1 minute and 30 seconds? >>> datetime.time(minute=1.5) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: integer argument expected, got float >>> datetime.time(second=90) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: second must be in 0..59 I'd like to use datetime.time for time conversions like this. It seems silly to have to do all the math and sanitize the inputs manually. If I have to do all that work myself, then why use this module at all? Is there another module or function that I am missing? Thanks, Ben

2 1

Expose PyFloat_AsDouble to Python
by Mark Dickinson May 29, 2020

May 29, 2020

Here's the short version: * "Convert float-like object to a float" is a useful operation * For a C extension, there's an obvious way to spell that operation: use `PyFloat_AsDouble` * In pure Python, there's no Obvious Way To Do It * Proposal: expose the equivalent of `PyFloat_FromDouble(PyFloat_AsDouble(obj))` to Python * Question: where should it go? Longer version: Various parts of CPython that expect a float will accept anything "float-like"; examples include most of the `math` module functions, the `struct` module, `ctypes`, `statistics`, old-style string formatting with `"%f"` and the like, anything that uses `PyArg_ParseTuple` with the `'d'` format converter, anything that uses argument clinic and declares an input as `double`, and more. This is a Good Thing: it means that as a Python user, you can make your own types compatible with these operations simply by giving them a `__float__` method. Now suppose that as a Python user and 3rd party library writer, I want to write my own function that similarly accepts anything float-like, following the same rules that core Python uses. If I'm writing a C extension module, this is trivially easy: I use `PyFloat_FromDouble`, which is part of the stable ABI. However, if I'm writing in pure Python, it's surprisingly awkward to clearly express the equivalent behaviour, and I'd like to have an obvious way to do it. Some of the non-obvious ways: 1. Use the `float` constructor. This _ought_ to be the one obvious way to do it: it'll certainly accept anything float-like and convert it to a float. But it will also accept instances of `str`, `bytes`, `bytearray`, and anything supporting the buffer protocol, so if you want to exclude those (for example, because passing a non-number to your numeric code is likely to be a coding error, and you'd like to catch it as such) you need to do a LBYL check. 2. Call the object's `__float__` method. But this is fraught with peril, too: for a proper equivalent, you need to be careful to look up `__float__` on the type, not the object itself. And then a new version of Python changes `PyFloat_AsDouble` to also accept objects with `__index__`, and suddenly your version no longer matches what Python does. (This happened.) 3. Leverage the `math` module's existing ability to do this. For example, I can do `x_as_float = math.copysign(x, x)`. This works, giving me _exactly_ the semantics that I want. But I think it would be stretch to call this an *obvious* way to do it. 4. Write your own small C extension containing a single function that looks like `return PyFloat_FromDouble(PyFloat_AsDouble(obj))` (with extra error checking, fast path for exact floats, etc.). So my modest proposal is: expose the conversion represented by `PyFloat_AsDouble` to Python somewhere: either on the `float` type itself, or somewhere in the standard library. My question for everyone on this list is: _if_ it were to be added, where should it go? (There's also the "what should it be called" question, of course.) I have a proof-of-concept PR [1] that exposes this in the `operator` module as `operator.as_float`. See also the discussion on the tracker [2]. [1] https://github.com/python/cpython/pull/20481 [2] https://bugs.python.org/issue40801

3 3

An HTTP API to list versions of Python
by Julien Palard May 29, 2020

May 29, 2020

We do not have a standard way to get (programatically) informations about Python versions, this leads to having to commit to numerous places during each releases, and some other places never get the information (like old docs, see [1]). Maybe we should provide an HTTP API listing Python versions, like: { "current_stable": "3.8.3", "versions": [ { "branch": "master", "status": "dev", "version": "3.10", "release_manager": "Łukasz Langa", "doc_languages": ["en", "es", "fr", "ja", "ko", "pt-br", "zh-cn"] }, { "branch": "3.9", "latest": "3.9.0b1", "status": "features", "pep": 596, "version": "3.9", "release_manager": "Łukasz Langa", "doc_languages": ["en", "fr", "ja", "ko", "pt-br", "zh-cn"] }, { "branch": "3.8", "latest": "3.8.3", "status": "bugfix", "pep": 569, "version": "3.8", "release_manager": "Łukasz Langa", "first_release": "2019-10-14", "end_life": "2024-10", "doc_languages": ["en", "fr", "ja", "ko", "pt-br", "zh-cn"] }, { "branch": "3.7", "latest": "3.7.7", "status": "bugfix", "pep": 537, "version": "3.8", "release_manager": "Ned Deily", "first_release": "2018-06-27", "end_life": "2023-06-27", "doc_languages": ["en", "fr", "ja", "ko", "pt-br"] } ... (Don't really mind this particular example, it's a random first though, the doc_languages should probably be a dict instead, with links to the actual docs, ...) It could be used to (and maybe avoid a commit) to: - build the documentation version switcher and languages switchers - toggle the red deprecated banner (to be seen on [2] if you missed it) - generate the versions table at https://devguide.python.org/ - generate the Docs by version menu - generate the Download menu and download page of python.org - list the documentations versions to be built by docsbuild-script - probably many scripts could use it too, I can think of [3] - ... It could become more detailed about each minor versions, git tag, links to changelogs, links to the repositories, to the docs, download links, and so on. Maybe this idea will just add a commit to do between the numerous commits to already do during a release... (à la https://xkcd.com/927/), and a SPOF. What do you think? [1]: https://mail.python.org/archives/list/docs@python.org/thread/L724YZXXHQZQP4… [2]: https://docs.python.org/2.7/ [3]: https://gitlab.com/python-devs/ci-images/-/blob/master/Dockerfile#L19 -- [Julien Palard](https://mdk.fr)

8 9

Re: Python __main__ function
by Jose Veiga May 29, 2020

May 29, 2020

While is possible to use `if __name__ == '__main__':` several times in the same script, your proposed magic function `def __main__()` cannot be redefined. Not to speak about lexical scope differences between one approach and the other. So your proposal is reducing features instead of expanding them. Jose Veiga

6 6

Re: Python __main__ function
by Chris Angelico May 28, 2020

May 28, 2020

On Fri, May 29, 2020 at 5:54 AM <tritium-list(a)sdamon.com> wrote: > > > -----Original Message----- > > From: Chris Angelico <rosuav(a)gmail.com> > > > > People can already put all their main logic into a function. If you want > to unit- > > test your main function, that's the best way to do it. > > The trouble is, how much goes into main() and how much into if __name__ > > == '__main__'? For example: Does your main function accept a list of > > arguments, or does it look in sys.argv? Neither answer is wrong. And that > > means the best way is to NOT force everyone across all of Python to choose > > the same way - let people write their code their way. > > > > People write main entry points that are not exactly this? > > If __name__ == '__main__': > sys.exit(main(sys.argv[1:])) > Of course they do! Much more common in my code is this: if __name__ == "__main__": main() or some variation that fetches sys.argv but doesn't call sys.exit. And quite a number of my projects include a global try/except around main() to log errors to a file as well as to stderr (due to where they're used), or process their args individually (for arg in sys.argv[1:]: do_stuff(arg)), or some other variation. It's most certainly NOT always exactly what you show there. And that's the problem. If it's part of the language definition, it will always have *exactly* the same semantics. As an idiom, it can be varied slightly to suit the situation, but as a language feature, it's rigid. ChrisA

1 0

Allowing assertEqual etc to take multiple items
by Ram Rachum May 28, 2020

May 28, 2020

Hi guys! I'm a unittest newbie, so it's possible there's a better solution than what I'm proposing here. If there is, please let me know. I recently submitted this PR <https://github.com/more-itertools/more-itertools/pull/430/files>, and I had to use assertSequenceEqual twice, just because I wanted to compare 3 sequences to each other rather than 2. Wouldn't it be possible to make assertEqual and its entire family (including assertSequenceEqual and the dozen of other methods) accept an arbitrary number of arguments, and compare all of them to each other? That way I could do this: self.assertSequenceEqual(sliced_tuple_0, sliced_tuple_1, sliced_range) We could discuss whether they'll do the faster version of a == b == c, or a more extensive check between every 2 items individually. What do you think?

4 4

Re: Optional keyword arguments
by Chris Angelico May 28, 2020

May 28, 2020

On Thu, May 28, 2020 at 10:42 AM Rob Cliffe <rob.cliffe(a)btinternet.com> wrote: > > > > On 26/05/2020 20:03, Chris Angelico wrote: > > On Wed, May 27, 2020 at 2:51 AM Steven D'Aprano <steve(a)pearwood.info> wrote: > >> But doing otherwise, having Undef be *not an object* but a kinda ghost > >> in the interpreter, is a huge language change and I doubt it would be > >> worth it. > >> > > But is it a huge change? I thought so too, until Greg suggested a > > quite plausible option: leave the local unbound. There'd be two > > changes needed, and one of them could have other value. The semantics > > would be exactly the same as any other unbound local. > > > > def foo(): > > if False: x = 0 > > # what is x now? > > > > There is no *value* in x, yet x has a state. > > [snip] > Sorry, I don't think it does. It's not in locals(), it doesn't exist > anywhere in memory. > It might be created later in foo(), but it would then have a value. It is in the state of being unbound, and you can prove this by attempting to print it out. >>> foo() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 3, in foo UnboundLocalError: local variable 'x' referenced before assignment This is NOT the same as if the assignment hadn't been in the code (which would have gone looking for a global or builtin x). So yes, it does have a state. ChrisA

1 0

len(path) == len(str(path))
by Ram Rachum May 25, 2020

May 25, 2020

Hi everyone, I submitted a PR today, and Serhiy decided it needs a discussion on python-ideas and agreement from core developers before it could go forward. BPO: https://bugs.python.org/issue40752 PR: https://github.com/python/cpython/pull/20348 Today I wrote a script and did this: sorted(paths, key=lambda path: len(str(path)), reverse=True) But it would have been nicer if I could do this: sorted(paths, key=len, reverse=True) So I implemented `PurePath.__len__` as `str(len(path))`. Serhiy and Remi objected, because it might not be obvious that the length of the path would be the length of string. What do you think? Can I get some +1s and -1s for this change? Thanks, Ram.

23 46

type hints : I'd like to suggest allowing unicode → as an alternative to ->
by Thierry Parmentelat May 24, 2020

May 24, 2020

well it’s all in the title the specific character that I am referring to is this one In [1]: print("\u2192”) → https://unicode-table.com/en/2192/ —— just curious about how people would feel about taking better advantage of non-ascii characters when that seems to make sense fyi here’s how both options appear in a markdown-based website thanks !

24 68