Mailman 3 March 2019 - Python-ideas

Simpler thread synchronization using "Sticky Condition"
by Richard Whitehead March 27, 2019

March 27, 2019

Problem: Using Python's Condition class is error-prone and difficult. For example, the following pseudo-code would be quite typical of its use: condition = threading.Condition() def sender(): while alive(): wait_for_my_data_from_hardware() with condition: send_data_to_receiver() condition.raise() def receiver(): while alive(): with condition: condition.wait() receive_all_data_from_sender() process_data() Main code will then run sender() and receiver() in separate threads. (I know that in a simple case like this, I could just use a Queue. I've been working with code that needs to service several events, where polling queues would introduce overhead and latency so a Condition must be used, but I've simplified the above as much as possible to illustrate the issues even in this simple case). There are two issues even with the above case: 1. Raising a condition only has any effect if the condition is already being waited upon; a condition does not "remember" that it has been raised. So in the example above, if the receiver starts after the sender, it will wait on the condition even though data is already available for it. This issue can be solved by rearranging the code, waiting at the end of the loop rather than the start; but this is counter-intuitive and lots of examples online look like the above. 2. In the receiver, the condition has to be kept locked (be inside the "with" statement") all the time. The lock is automatically released while the condition is being waited upon, but otherwise it must be locked, otherwise we risk missing the condition being raised. The result is that process_data() is called with the lock held - and so this will prevent the sender going round its loop. The sending thread becomes a slave to the processing thread. This may have a huge performance penalty, losing the advantage of loose coupling that threading should provide. You might think that using an Event, rather than a Condition, would solve things, since an Event does "remember" that is has been set. But there is no atomic way to wait on an event and clear it afterwards. Solution: The solution is very simple: to create something that might be called a StickyCondition, or an AutoResetEvent. This is a Condition that does remember it has been raised; or if you prefer, it is an Event that resets itself after it has been waited upon. The example then becomes: auto_event = threading.AutoResetEvent() def sender(): while alive(): wait_for_my_data_from_hardware() send_data_to_receiver() auto_event.set() def receiver(): while alive(): auto_event.wait() receive_all_data_from_sender() process_data() This solves both of the issues described: the receiver will not wait if data is already available, and the sender is not blocked by a lock that is being held by the receiver. The event doesn't need to be locked at all, because of its internal memory. Implementation is trivial, involving a boolean and a Condition in the cross-thread case. I presume it would be more involved for the cross-process case, but I can see no reason it would be impossible. Please let me know if you think this would be useful.

4 8

singledispatch for methods
by Tim Mitchell March 27, 2019

March 27, 2019

Hi All, functools.singledispatch does not work on methods. There are 2 packages that do this for methods one on GitHub only https://gist.github.com/adamnew123456/9218f99ba35da225ca11 and my pypi package https://pypi.org/project/methoddispatch/. There are also a couple of stack overflow posts about it ( https://stackoverflow.com/questions/24601722/how-can-i-use-functools-single… , https://stackoverflow.com/questions/24063788/python3-singledispatch-in-clas… ) with about 60 votes between them. Is it time to add singledispatch for methods to the core library? If so, would the methoddispatch implementation suffice or are there changes you would like made? Thanks for your time.

3 2

Re: [Python-ideas] singledispatch for methods
by Tim Mitchell March 27, 2019

March 27, 2019

The motivation is the same as for functools.singledispatch ( https://www.python.org/dev/peps/pep-0443/) - provide generic methods. These are useful for visitor implementations that are cleaner than the visit_<class_name> approach typically used today (e.g. https://github.com/mbr/visitor). It is also useful for writing serialisers and encoders. The motivation is also that people keep using the method name munging approach (e.g. mypy) because functools.singledispatch only works on functions and not class methods. Cheers Tim On Wed, Mar 27, 2019 at 4:33 PM Michael Selik <michael.selik(a)gmail.com> wrote: > On Tue, Mar 26, 2019, 1:09 PM Tim Mitchell <tim.mitchell.chch(a)gmail.com> > wrote: > >> Is it time to add singledispatch for methods to the core library? >> > > What's the motivation for it, beyond the fact that it's possible? > > Regarding jargon, aren't Python's instance methods are already > single-dispatch, because they receive the instance as the first argument? > >>

1 0

Allow not in lambda expressions
by Abdur-Rahmaan Janhangeer March 27, 2019

March 27, 2019

Suppose i have (lambda x: x if x != None else '')(someVar) returning an empty string if none but, if "not" was allowed (lambda x: x if x not None else '')(someVar) it might have been more elegant PROPOSAL: Allow "not" in lambda expressions -- Abdur-Rahmaan Janhangeer Mauritius <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> Garanti sans virus. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

3 4

A directive for indentation type, stricter indentation parsing.
by Mikhail V March 26, 2019

March 26, 2019

Not a proposal yet, but some thoughts: I think it would help in a longer perspective if a user could include a directive in the header of the source code file that defines indentation character(s) for this source file. So this source would be parsed strictly by this char (or sequence). E.g.: # indent "\t" ... Would force the Python parser to use exactly 1 tab for 1 indent level. # indent " " ... Would accordingly force the parser to use exactly 4 spaces for 1 indent level. Frankly I don't have much proof in hand for that will be a good addition, but intuitively I suppose it would help with some possible future features and in general, ease the development of source processors. One possible example: if a potential future feature would require a statement, and moreover require it to be indentation-aware? Lets take e.g. a multi-line string: s = """ Hello world """ print (s) >>> Hello world Here it is not so intuitive (unless you already know) how the string would be parsed (given that Python blocks are actually indentation-based). So if one would want to try introduce a new indent-aware string type and look into possible parsing disambiguation scenarios - it will be not an easy task. E.g. say one proposes a syntax for auto-unindented string block: s = !!! Hello world print (s) >>> Hello world (no leading newline, no leading indent in resulting string, which is a bit more expected result IMO). Then it seems one can define the parsing rule unambiguously _only_ if one has a strictly defined character sequence for the indent level (e.g. 1 tab or 4 spaces, but not both). Thus it seems such a directive would be a prerequisite for such feature. And in general, I think it could help to make automatic conversions from one type of indentation to other easier. Mikhail

5 6

Re: [Python-ideas] Simpler thread synchronization using "Sticky Condition"
by Richard Whitehead March 26, 2019

March 26, 2019

Antoine, Thanks for a couple of very useful comments. The usefulness of wait_for(), especially in avoiding the problem of spurious-waits, is something I'd forgotten - this is really just syntactic sugar, but I agree it is very useful - as illustrated by the fact that I didn't remember to put my wait() call in its own loop in my example! On the other hand, the lack of locking makes this new thing more like an Event than a Condition. Also, since it would only make sense for one thread to wait on this object, rather than many, that again makes it more like an Event. And Event only has wait(), not wait_for() - it makes no sense to "loop" on an object that is "sticky", you would just get livelock. So in summary, this new class (or behaviour) would still only have wait() and not wait_for(). Does that devalue it, do you think? I know that C# has AutoResetEvent, but I didn't know it was native to the Windows API. I'm pretty sure it is not native to Linux (there are a couple of other useful features, such as "Wait on multiple events", that are in Windows but not Linux, and therefore are also not in Python). Does this affect implementation - would you expect an AutoResetEvent to be implemented natively on Windows, rather than being composed out of pure Python using a boolean and a Condition? Regarding the link you sent, I don't entirely agree with the opinion expressed: if you try to use a Semaphore for this purpose you will soon find that it is "the wrong way round", it is intended to protect resources from multiple accesses, not to synchronize those multiple accesses. Thanks, Richard -----Original Message----- From: python-ideas-request(a)python.org Sent: Tuesday, March 26, 2019 10:49 AM To: python-ideas(a)python.org Subject: Python-ideas Digest, Vol 148, Issue 143 Send Python-ideas mailing list submissions to python-ideas(a)python.org To subscribe or unsubscribe via the World Wide Web, visit https://mail.python.org/mailman/listinfo/python-ideas or, via email, send a message with subject or body 'help' to python-ideas-request(a)python.org You can reach the person managing the list at python-ideas-owner(a)python.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Python-ideas digest..." Today's Topics: 1. The Mailing List Digest Project (Abdur-Rahmaan Janhangeer) 2. Simpler thread synchronization using "Sticky Condition" (Richard Whitehead) 3. Re: Simpler thread synchronization using "Sticky Condition" (Antoine Pitrou) 4. Re: A directive for indentation type, stricter indentation parsing. (Mikhail V) ---------------------------------------------------------------------- Message: 1 Date: Tue, 26 Mar 2019 09:07:15 +0400 From: Abdur-Rahmaan Janhangeer <arj.python(a)gmail.com> To: Python <python-list(a)python.org>, python-ideas <python-ideas(a)python.org> Subject: [Python-ideas] The Mailing List Digest Project Message-ID: <CADrxXXmP3-1eyYPNRgaQb1MOcXA8yYVHxxDk6orX85gExR4wRg(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" As proposed on python-ideas, i setup a repo to turn mail threads into articles. here is the repo https://github.com/Abdur-rahmaanJ/py-mailing-list-summary i included a script to build .md to .html (with syntax highlighting) here is the index https://abdur-rahmaanj.github.io/py-mailing-list-summary/ included 3 articles as a start if you want to contribute an article, just follow existing .md format and put it in the .md folder planning to go across ideas, list and dev i can tell you, it's a really enjoyable experience. psst. we can enhance some html later -- Abdur-Rahmaan Janhangeer Mauritius

1 0

Why not ['a','b','c'].join(',') ?
by Juancarlo Añez March 25, 2019

March 25, 2019

I know it has been discussed endlessly, so just a gentle reminder about the final arguments would be good. I think I remember it was discussed recently, mentioning that join() doesn't convert elements to strings? This came up while reading this speculative article about how programmers migrate from one programming language to the other (I call it speculative because there's no hard data, but the conclusions and comments pretty much match my experience and my observations over decades [I would add an arrow from Python3 to Rust]). https://apenwarr.ca/log/20190318 In that sense, I think it wouldn't hurt to make Python more familiar/friendly to people coming from other languages, even if it breaks "There should be one-- and preferably only one --obvious way to do it." -- Juancarlo *Añez*

10 18

I'm saying goodbye for a bit
by Jonathan Fine March 25, 2019

March 25, 2019

Hi I've been active recently in some threads, that have become a bit heated. To help things cool down, I won't be posting for a while. I don't know how long, but certainly not until Wednesday 3 April. You can of course contact me off-list if you want. -- Jonathan

1 0

Improve os.times() resolution
by Giampaolo Rodola' March 24, 2019

March 24, 2019

It turns out we could use resource.getrusage() which provides micro seconds (tested on Linux and macOS): import os, resource for x in range(10000000): # warm up pass for x in range(5): a = os.times() b = resource.getrusage(resource.RUSAGE_SELF) print(a.user, a.system) print(b.ru_utime, b.ru_stime) ...it prints: 0.39 0.01 0.394841 0.011963999999999999 0.39 0.01 0.394899 0.011966 0.39 0.01 0.394908 0.011966 0.39 0.01 0.394936 0.011967 0.39 0.01 0.394963 0.011968 getrusage(RUSAGE_CHILDREN) can be used to calculate "children_user" and "children_system". I see 2 possibilities here: 1) doc fix, mentioning that resource.getrusage provides a better resolution 2) if available (it should always be as it's a POSIX standard), just use getrusage in Modules/posixmodule.c. It seems we can check availability by reusing HAVE_SYS_RESOURCE_H and HAVE_SYS_TIME_H definitions which are already in place. I'm not sure what's best to do as os.* functions usually expose the original C function with the same name, but given that "elapsed" field is not part of times(2) struct and that on Windows "elapsed", "children_user" and "children_system" are set to 0 it appears there may be some space for flexibility here. Thoughts? -- Giampaolo - http://grodola.blogspot.com

3 3

Add subprocess.Popen suspend() and resume()
by Giampaolo Rodola' March 24, 2019

March 24, 2019

Hello, I've been having these 2 implemented in psutil for a long time. On POSIX these are convenience functions using os.kill() + SIGSTOP / SIGCONT (the same as CTRL+Z / "fg"). On Windows they use undocumented NtSuspendProcess and NtResumeProcess Windows APIs available since XP. The same approach is used by ProcessHacker and - I suppose - pssupend.exe, both from Sysinternals team. It must be noted that there are 3 different ways to do this on Windows (https://stackoverflow.com/a/11010508/376587) but NtSuspend/ResumeProcess appears to be the best choice. Possible use case: <<PsSuspend lets you suspend processes on the local or a remote system, which is desirable in cases where a process is consuming a resource (e.g. network, CPU or disk) that you want to allow different processes to use. Rather than kill the process that's consuming the resource, suspending permits you to let it continue operation at some later point in time.>> Thoughts? -- Giampaolo - http://grodola.blogspot.com

4 5