Mailman 3 June 2020 - Python-ideas

Shared Semaphores for synchronisation across unrelated processes
by Vinay Sharma 09 Jun '20

09 Jun '20

Hi, Python has integrated shared memory into standard library starting from 3.8 (https://docs.python.org/3/library/multiprocessing.shared_memory.html <https://docs.python.org/3/library/multiprocessing.shared_memory.html>), which provides a user friendly API to access shared memory across unrelated processes using names. But, there are no synchronisation mechanisms present in the standard library to prevent race conditions when shared memory is accessed across unrelated processes. I had earlier created an enhancement issue <https://bugs.python.org/issue38035> at bugs.python.org <http://bug.python.org/>, which contains more detailed discussion on the same. I am posting this here after a suggestion from a Python contributor. Feedback on the same will be very helpful.

4 11

Re: Add close method to queue
by Matthew Einhorn 09 Jun '20

09 Jun '20

Accidentally responded off list. On Tue, Jun 9, 2020 at 1:06 PM Андрей <heckad(a)yandex.ru> wrote: > Thank you for your comment. > > > > This feature is very needed for > https://github.com/mosquito/aio-pika/pull/327. > > > > PS How to send response on this message? I don’t see this message on *mail.python.org > <http://mail.python.org>. * > You need to register on python's bug tracker if you want to be able to respond to the bug report there. > And where do you find RLock? > It's in the threading package <https://docs.python.org/3.9/library/threading.html#threading.RLock>. You can use a normal Lock, it's just that I needed to be able to call this from __del__ <https://docs.python.org/3/reference/datamodel.html#object.__del__> so I needed a lock that can be re-entered. > > > Отправлено из приложения "Почта > <https://go.microsoft.com/fwlink/?LinkId=550986>" для Windows 10 > > > > *От: *Matthew Einhorn <moiein2000(a)gmail.com> > *Отправлено: *9 июня 2020 г. в 19:41 > *Кому: *Kazantcev Andrey <heckad(a)yandex.ru> > *Тема: *Re: [Python-ideas] Add close method to queue > > > > I was just looking for something similar, but this was already requested > and rejected for lack of use cases here: > https://bugs.python.org/issue29701. > > > > For my own use case, a RLock with a class variable indicating the queue > was closed did the trick. > > > > On Tue, Jun 9, 2020 at 8:25 AM Kazantcev Andrey <heckad(a)yandex.ru> wrote: > > I have a problem with notifying all current subscribers and new > subscribers about the closure of the queue and the reason. For example, I > have a producer that reads messages from websocket or something else and > send this to a queue, and several consumers (I do not know how many). If > any exception occurred, then all current subscribers and subscribers which > will be added later should know about this error. I tried to send an > exception to a queue, but that did not help, because I have several > consumers. Also, this will not protect new consumers. I propose to add a > new close method with exc argument, which will throw an exception when > calling the get method, and also throw an exception for all current > _getters. > _______________________________________________ > Python-ideas mailing list -- python-ideas(a)python.org > To unsubscribe send an email to python-ideas-leave(a)python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/EYVGO… > Code of Conduct: http://python.org/psf/codeofconduct/ > > >

1 0

Add close method to queue
by Kazantcev Andrey 09 Jun '20

09 Jun '20

I have a problem with notifying all current subscribers and new subscribers about the closure of the queue and the reason. For example, I have a producer that reads messages from websocket or something else and send this to a queue, and several consumers (I do not know how many). If any exception occurred, then all current subscribers and subscribers which will be added later should know about this error. I tried to send an exception to a queue, but that did not help, because I have several consumers. Also, this will not protect new consumers. I propose to add a new close method with exc argument, which will throw an exception when calling the get method, and also throw an exception for all current _getters.

2 1

Augment abc.Set API (support named set methods for dictionary view objects)
by ava＠yert.pink ava＠yert.pink 07 Jun '20

07 Jun '20

Currently the dictionary view objects behave exactly like sets when set operations defined on them are used, such as the pipe or ampersand operators: >>> d.keys() | [1] {1, 2} dictionary views are even sets by the ABC standard, having support for all the required abstract methods: >>> isinstance(d.keys(), collections.abc.Set) True However, the dict views do not have the equivalent named set methods, such as `intersection` or `issuperset`. It seems that the collections.abc.Set API was purposely crafted for returning True when a dict view is given as an argument to isinstance, as c urrently, the `isdisjoint` method is the only named method that the API requires. I propose that the `Set` ABC API should be augmented to contain all of the named methods. This would provide consistency in the collections, and enhance the duck typing capabilities of the `Set` abc.

7 12

PYTHONLOGGING env variable
by Bar Harel 06 Jun '20

06 Jun '20

Another idea I've had that may be of use: PYTHONLOGGING environment variable. Setting PYTHONLOGGING to any log level or level name will initialize logging.basicConfig() with that appropriate level. Another option would be that -x dev or a different -x logging will initialize basic config. Will be useful mostly for debugging purposes instead of temporarily modifying the code. Kinda surprised it doesn't exist tbh. Bar Harel

9 22

Advanced Debugger Support C-API is incomplete?
by Andrew Pashkin 06 Jun '20

06 Jun '20

I want to implement my own version of _PyThread_CurrentFrames() <https://github.com/python/cpython/blob/e838a9324c1719bb917ca81ede8d766b5cb5…>, but it seems like it's not possible using public C-API. The comment <https://github.com/python/cpython/blob/e838a9324c1719bb917ca81ede8d766b5cb5…> in the source code of _PyThread_CurrentFrames() says that lists of interpreters and threads can be mutated even when GIL is held and so HEAD_LOCK() should be used. At the same time Advanced Debugger Support <https://docs.python.org/3/c-api/init.html#advanced-debugger-support> API exposes access to these lists but there is no way (or I don't see how) to achieve the same thing that HEAD_LOCK() does. Am I missing something or Advanced Debugger Support is incomplete? -- With kind regards, Andrew Pashkin. cell phone - +375 (44) 492-16-85 Skype - waves_in_fluids e-mail - andrew.pashkin(a)gmx.co.uk

1 1

Automatic notification when your PR test fails or succeeds
by Ram Rachum 05 Jun '20

05 Jun '20

Hi guys, I'm not sure whether this is the right mailing list for this, because it's more for the Python development process on GitHub than Python itself. If this belongs somewhere else, let me know. When you open a PR to CPython, it would be cool if there was an option to get email notifications on whether the tests fail or succeeded. For example, I may open a PR as a draft, and only after I get notification on success, mark it as "Ready for review". Right now I just poll it manually, which is annoying. What do you think?

3 2

asyncio: recursive event loops
by Celelibi 04 Jun '20

04 Jun '20

Hello world, I've been recently trying to switch from a library using gevent to a library using asyncio. And do so without contaminating the whole code with the "async" keyword in one go. As expected, calling the synchronous code from a coroutine is pretty straightforward. But the other way around is a bit more tricky. If the function is not running in the thread that runs the event loop, I can use something like this: asyncio.run_coroutine_threadsafe(...).result() Which seems to work, despite the documentation saying that "If the future's result isn't yet available, raises InvalidStateError.". But if the function that want to call a coroutine is in the thread that runs the event loop, then there's not much I can do to run a coroutine in the event loop that indirectly called that function. As a somewhat more concrete example, let's consider a class that sends and receive messages: class Client: async def on_msg(self, msg): process(msg) async def asend(self, msg): print("async sending", msg) return len(msg) def send(self, msg): print("sending", msg) # return await self.asend(msg) async def run(self): await self.on_msg("PING") def process(msg): # Some legacy code that indirectly ends up wanting to call: print("Received", msg) print(client.send("PONG")) client = Client() asyncio.run(client.run()) Unless I missed something, it's pretty difficult to have `Client.send` "await" on `Client.asend` without contaminating the whole code with "async" and "await". I only see two solutions right now. Either have `Client.on_msg` call `process` in a new thread and then use the trick above. But running `process` in a separate thread might have some consequences since it doesn't expect it. Or have `Client.send` start a new event loop in a new thread. Which might behave suprisingly when sharing file descriptors with the existing loop. What I'm proposing is to change a bit the behavior of `loop.run_until_complete` as follow: - If the loop isn't running and there's no loop in the current context, keep the current behavior. - If the loop is running, but not in the current context, use the `call_soon_threadsafe` trick above. - If the loop is running in the current context, then loop until the coroutine is done. Additionally, `asyncio.run` could be changed similarly. There's of course a strong caveat that it's easy to create an infinite recursion with this. But I think it's worth it as it would allow a much easier integration of asyncio into existing code. Which (I think) limit its adoption. What do you think about it? Best regards

1 0

Optional keyword arguments
by James Lu 02 Jun '20

02 Jun '20

Many a python programmer have tired to see code written like: def bar(a1, a2, options=None): if options is None: options = {} ... # rest of function syntax if argument is not passed, evaluate {} and store to options def foo(options:={}): pass syntax if argument is not passed or is None, evaluate {} and store to options* def foo(options?={}): pass The Zen of Python states "there shouldn't be two ways to do the same thing." Thus, only one of ":=" or "?=" should be adopted. They should be evalued on: - Which encourages writing better code? - Which catches mistakes easier? Do we want to encourage callers to pass None to indicate default arguments? spam = { data: True } if arg else None bar(a1, a2, param=spam) versus bar(a1, a2, { data: True }) if arg else bar(a1, a2) versus _ = foo.curry(a1, a2) _({data: True}) if arg else _(a1, a2) Since Python is a strongly typed language, it seems more consistent to me that this code should throw an error: def getoptions(): ... # code to get options # whoops! missing return statement #return os.environ foo(a1, a2, param=getoptions()) := should be adopted because it catches mistakes more quickly. On the other hand, ?= replaces the "if kwarg is not None: kwarg = ..." idiom. (I propose adopting only ":=". I show "?=" as a strawman.)

23 151

How to propose a change with tests where the failing test case (current behaviour) is bad or dangerous
by Steve Barnes 02 Jun '20

02 Jun '20

There is a long standing issue, (or at least I consider it a poor feature some others consider it compliance with other existing tools), with a couple of python standard libraries that I would like to raise an issue for & PR(s) to address but when it comes to putting tests in place the failing test case can potentially crash the host machine, sometimes in an unrecoverable manner, due to the potential to generate an infinite size file. Is there a way of proposing a change with tests that refuse to run in some situations or can be terminated based on time or resource limits being exceeded? Steve Barnes

9 21