Mailman 3 December 2020 - Python-ideas

PEP 533 Redux?
by Paul Bryan 31 Dec '20

31 Dec '20

I've now encountered an issue I believe that PEP 533 was intended to address: When an asynchronous context manager is created within an asynchronous generator, if the generator is not iterated fully, the context manager will not exit until the event loop cancels the task by raising a CancelledError, long after the context manager is assumed to be out of scope. Per PEP 525, I can call aclose coroutine method to cleanup the generator, but it requires the code iterating to be aware that that closing the generator is necessary. Any appetite for putting PEP 533 back on the table to address this issue?

2 2

Fixing and improving statement caching in sqlite3
by James Oldfield 30 Dec '20

30 Dec '20

Hi Python-Ideas, I was browsing the source code to the sqlite3 standard library module (as one does over the Christmas holidays) and was surprised to find that the eviction mechanism for cached prepared statements is fairly broken ... and has been for the last 15 years! This is just from reading the source code, so hopefully I've not made a fool of myself by misreading it. I've reported the problem to the upstream pysqlite repository: https://github.com/ghaering/pysqlite/issues/141 This got me thinking about potential fixes and improvements, which I thought might be worth discussing here, especially as maybe not all changes from that repo are still merged into mainline Python. 1. My first suggestion is to replace the odd cache logic with a straightforward classic LRU cache. I'm hoping this is uncontroversial, but I can imagine there might be programs that somehow depend on the current behaviour. But my suspicion is that many more suffer and just no one looked hard enough to notice. 2. My next suggestion is to allow disabling the cache (removing it entirely isn't an option for backwards compatibility) and instead cache one statement per cursor so long as consecutive calls to `execute()` or `executemany()` use identical SQL. This neatly avoids changing the interface while still allowing devs complete control over how they cache statements: you can put them in your own LRU cache structure, or cache specific ones by reusing just those cursors, or do anything else that you like. This API would also cleanly mirror how prepared statements work in the underlying SQLite C library. My first thought was to have a separate class representing a prepared statement, and allow that to be used as a factory for creating cursors; but SQLite prepared statements can only be executed and iterated once at a time before being reused, so using the Cursor object itself for caching the statement makes more sense. 3. My final suggestion is a small backwards-compatible change to the interface to make the previous idea particularly clear and easy to use: allow the SQL parameter to `execute()` and `executemany()` to be omitted (or None) to mean the same query as last used on that cursor, which guarantees it will use the cached prepared statement. Even better, also allow SQL to be passed in when creating the cursor. This leads to particularly straightforward statement reuse code: conn = sqlite3.connect(path_to_db, isolation_level=None) cur = conn.cursor(sql="SELECT * FROM Foo WHERE i > ?") print(cur.execute(parameters=(3,)).fetchall()) for row in cur.execute(parameters=(2,)): print(row) What do you think? Thanks, James

2 1

Add aggregations and joins code-generating library along with itertools
by Nikita Almakov 29 Dec '20

29 Dec '20

Hello everyone! The idea: it would be cool to have the functionality of "convtools" library available along with "itertools". Python should provide decent functionality of aggregations and joins out-of- the-box. Given that named tuples generate some could, this one could too: https://pypi.org/project/convtools/ https://convtools.readthedocs.io/en/latest/cheatsheet.html https://github.com/iTechArt/convtools Please, let me know your view on this or any ideas (I would appreciate it). Thank you and happy holidays! Nikita Almakov

3 4

str.isfloat()
by Tushar Sadhwani 28 Dec '20

28 Dec '20

str currently has methods like isdecimal, isnumeric and isdigit, but there isn't an isfloat check, which could be very handy. a trivial implementation could be as simple as: try: float(self) return True except ValueError: return False

5 7

asyncio.Pipe()
by Roger Iyengar 28 Dec '20

28 Dec '20

I believe that asyncio should have a way to wait for input from a different process without blocking the event loop. The Asyncio module currently contains a Queue class that allows communication between multiple coroutines running on the same event loop. However, this module is not threadsafe or process-safe. The multiprocessing module contains Queue and Pipe classes that allow inter-process communication, but there's no way to directly read from these objects without blocking the event loop. I propose adding a Pipe class to asyncio, that is process-safe and can be read from without blocking the event loop. This was discussed a bit here: https://github.com/python/cpython/pull/20882#issuecomment-683463367 This could be implemented using the multiprocessing.Pipe class. multiprocessing.connection.Connection.fileno() returns the file descriptor used by a pipe. We could then use loop.add_reader() to set an asyncio.Event when something has been written to the pipe by the other process. I did this all manually in a project I was working on. However, this required me to learn a considerable amount about asyncio. It would have saved me a lot of time if there was an easy documented way to wait for input from another process in a non-blocking way. One compelling use case for this is a server that uses asyncio, which receives inputs from clients, then sends these to another process that runs a neural network. The server then sends the client a result after the neural network finishes. ProcessPoolExecutor does not seem like a good fit for this use case, because the process needs to stay alive and be re-used for subsequent requests. Starting a new process for each request is impractical, because loading the neural network into GPU memory is an expensive operation. See here for an example of such a server (however this one is mostly written in C++ and does not asyncio): https://www.tensorflow.org/tfx/guide/serving

2 4

fsync-on-close io object
by Michael A. Smith 26 Dec '20

26 Dec '20

With all the buffering that modern disks and filesystems do, a specific question has come up a few times with respect to whether or not data was actually written after flush. I think it would be pretty useful for the standard library to have a variant in the io module that would explicitly fsync on close. You might be tempted to argue that this can be done very easily in Python already, so why include it in the standard io module? 1. It seems to me that it would be better to do this in C, so for the folks who need to make a consistency > performance kind of choice, they don't have to sacrifice any additional performance. 2. Having it in the io library will call attention to this issue, which I think is something a lot of folks don't consider. Assuming that `close` or `flush` are sufficient for consistency has always been wrong (at its limits), but it was less likely to be a stumbling block in the past, when buffering was less aggressive and less layered and the peak size and continuous-ness of data streams was a more niche concern. 3. There are many ways to do this, and I think several of them could be subtly incorrect. In other words, maybe it can't be done very easily and correctly after all. Providing "obviously right" ways to do things is the baileywick of the standard library, isn't it? Thanks for your consideration and feedback, Michael Smith

7 10

Allow return value in asynchronous generator
by Paolo Lammens 26 Dec '20

26 Dec '20

Currently, asynchronous generators don't allow a non-empty `return` statement: ```python import asyncio async def async_generator(): x = yield 1 await asyncio.sleep(1) if x: ... return "result A" # SyntaxError else: ... return "result B" ``` unlike regular generators which communicate their return value via the `value` attribute on the `StopIteration` exception: ```python def synchronous_generator(): x = yield 1 if x: ... return "result A" # raises StopIteration with value="result A" else: ... return "result B" ``` I think that `StopAsyncIteration` should behave likewise and have a `value` attribute containing the return value. I'm not aware of any reason to maintain this asymmetry.

1 1

Add decorator_with_params function to functools module
by Yurii Karabas 24 Dec '20

24 Dec '20

In the current python codebases, there are a lot of decorators with parameters, and their code in most cases contains a lot of code boilerplates. The purpose of this feature is to add `decorator_with_params` (I think the name can be changed to smth better) function to `functools` module. This function will allow using a decorated function as a simple decorator and decorator with parameter. I believe the real example will bring more context, for instance, `typing._tp_cache` function can be rewritten from: ``` def _tp_cache(func=None, /, *, typed=False): """Internal wrapper caching __getitem__ of generic types with a fallback to original function for non-hashable arguments. """ def decorator(func): cached = functools.lru_cache(typed=typed)(func) _cleanups.append(cached.cache_clear) @functools.wraps(func) def inner(*args, **kwds): try: return cached(*args, **kwds) except TypeError: pass return func(*args, **kwds) return inner if func is not None: return decorator(func) return decorator ``` To ``` @decorator_with_params def _tp_cache(func, /, *, typed=False): """Internal wrapper caching __getitem__ of generic types with a fallback to original function for non-hashable arguments. """ cached = functools.lru_cache(typed=typed)(func) _cleanups.append(cached.cache_clear) @functools.wraps(func) def inner(*args, **kwds): try: return cached(*args, **kwds) except TypeError: pass # All real errors (not unhashable args) are raised below. return func(*args, **kwds) return inner ``` And `_tp_cache` can be used as: ``` @_tp_cache(typed=True) def __getitem__(self, parameters): return self._getitem(self, parameters) ... @_tp_cache def __getitem__(self, parameters): return self._getitem(self, parameters) ``` Proposed implementation is quite simple (I used it in a lot of projects): ``` def decorator_with_params(deco): @wraps(deco) def decorator(func=None, /, *args, **kwargs): if func is not None: return deco(func, *args, **kwargs) def wrapper(f): return deco(f, *args, **kwargs) return wrapper return decorator ``` I believe it will be a great enhancement for the Python standard library and will save such important indentations. I have already opened an issue https://bugs.python.org/issue42455 (Sorry about that I didn't know that smth like python-ideas exists).

3 5

Should PEP 637 legalize "obj[]"?
by Brandt Bucher 24 Dec '20

24 Dec '20

I didn't participate in the mailing list discussions for PEP 637, but I have become somewhat involved in the reference implementation and final revisions. While reviewing the PEP, I noticed what I think is a missed opportunity to finally allow empty subscripting (`obj[]`). It seems to me to be a natural consequence of the `obj[*a]` and `obj[**k]` syntax which the PEP proposes, and I'm quite surprised that the PEP still outlaws it (the latest revision has language that says we're punting on this, but I'd like to change that). I'm wondering how others feel. Do you think we should allow `obj[]` as sugar for `obj[()]`, or leave it a `SyntaxError`? Steven, Stefano, and I are in agreement that we should at least consider allowing it. I didn't see it being discussed much at all (although I may be missing a thread or two). My primary motivation here is type-hinting generics (like `tuple[]`, currently spelled as `tuple[()]`). I'm worried that this could become more of an annoyance with the potential introduction of variadic generics (PEP 646) - not only for end-users who need to add type hints for empty generics, but also for library authors who will constantly need to special-case their reprs to turn `"MyType[]"` into `"MyType[()]"` or similar (I found that even mypy's error messages still don't do this correctly for `tuple` / `Tuple`!). But I don't really have much insight into how pervasive that issue would be, in practice. The way I see it, there are sort of four paths forward: - Add this to PEP 637. **I think this option makes the most sense.** - Add this to the new variadic generics PEP. - Propose this as its own PEP some time after 637 is accepted (sort of like PEP 618). - Don't pursue it, it's a bad idea. The only drawback I anticipate is beginners doing stuff with `some_mapping[]` and being surprised that it's adding/removing/looking up an empty tuple. But to me, `some_mapping[]` being the same as `some_mapping[()]` is no more confusing than `some_mapping[1]` being different from `some_mapping[1,]`. If anything, it's just a good teaching moment, where they can learn that positional indices are treated "as if" they are wrapped in parentheses. Or, for more advanced users, asking them what `x[y=z]` without the keyword argument would mean. Thoughts? Brandt

3 2

Re: Capture manylinux environment set-up in pyproject.toml
by lasizoillo 23 Dec '20

23 Dec '20

> > The manylinux project does not specify a way to record "environment set > up" information required to build wheels from certain packages. > > Some packages require that the stock manylinux environment be modified by > installing system dependencies, setting environment variables, modifying > the filesystem etc prior to building, or the building and bundling will > fail. Currently, package authors capture this information in project > specific documentation, or not at all. > > In the event that a wheel is not provided by the author, consumers that > want a compiled distribution (users focused on reproducible builds, build > efficiency) are required to locate this information in project specific > documentation or do research on the internals of the package to gather this > information themselves. Currently, package consumers can not easily modify > or contribute to this data as it is not standardized. > > What are everyone's thoughts about adding a section to pyproject.toml to > capture this information? This allows package consumers to easily find and > contribute back to this data, and programs like installers, builders etc > can read this data as well. Is this PEP-able or where could this best be > contributed? I am interested in (and intend on) contributing to solving > this problem if it is accepted as a good idea! Thank you for your input! > > Something like this: > > [manylinux1] > system_dependencies: > - foo-1.0.0 > environment_variables: > - FOO=BAR > python_packages: > - foo==1.0.0 > > Its interesting for me, but I see some problems. System_dependencies: many linux have different names (example: apache2 vs httpd). Sometimes same dependency is sliced in multiple packages (example: python3.9 depends on python3.9-minimal and...). How many times have you see a machine with python but without idle or tkinter? And naming convention is a pain for build systems dependencies and runtime system dependencies. Fortunately we have containers and can run different linux flavours. And old version of theese flavour which is very important. Environment variables: Usually used to inject configuration. My GOPATH env var is under my home. Maybe a env name can have a description, default value,... but not a fixed value, because is not easy to share with others. To avoid system dependencies, some build tools like https://github.com/joerick/cibuildwheel are centered on CI suites. With a CI you can use windows, linux, osx,... without think in mount virtual machines (libvirt, virtualbox,...) or coordinate buildbots farms more or less well configured. Once a wheel is compiled, build dependencies are no needed more. Then, cibuildwheel uses other libraries like auditwheel to embed shared libraries into wheels. So runtime system dependencies are included in wheel too. Mostly (see limitations https://github.com/pypa/auditwheel#limitations). So requirements shoud be somelike... * base_system_dependency: links to a OCI container, for example (open specifications if need to choose a technology). This mark minimal version of clib compatibility. * extra_base_system_repositories: We want modern libraries, with all patches, but running on old linux version to improve compatibility with clib. * build_dependencies: they must exists on base_system_dependencies or extra_base_system_repositories. ** build_system_dependencies: things used to test, compile and verify build. For example patchelf used by auditwheel ** build_python_dependencies: libraries used to build things. Like auditwheel used to pack into wheels runtime dependencies. * build_steps: Maybe even environment variables are a good idea if it will run in containers. It could include unit-tests, compilation, integration tests after compilation,... * artifacts: wheels, rpm packages, deb packages,... * runtime_dependencies: All things that can not be included in our generated artifacts. ** python_runtime_dependencies: The simple part ** system_runtime_dependencies: But for this section, system dependencies will not run containerized. Return to diffent names, available versions, incompatibilities,... I think that it's possible generate metadatas describing build requirements. Even if it fails for automated construction are a guide "TL;DR" of how install it. But if it can serve to generate automated constructions, repair metadatas will be easy and then automated construction will run. Then we'll need a pyproject2docker or pyproject2gitlabci or... Maybe this reference implementations should be needed to test specification. Metadatas should be used to avoid technological dependence in builds. If it no ends in a PEP is a good project to fill a private-repo with wheels. Regards, Javi

3 2