Mailman 3 March 2022 - Python-ideas

Re: [Python-Dev] An unambiguous way of initializing an empty dictionary and set
by Ethan Furman 14 Mar '22

14 Mar '22

On 3/13/22 14:49, joao.p.f.batista.97(a)gmail.com wrote: > Currently: > l = [] # new empty list > t = () # new empty tuple > s = set() # new empty set (no clean and consistent way of initializing regarding the others) <<< > d = {} # new empty dictionary > > Possible solution: > s = {} # new empty set > d = {:} # new empty dictionary (the ":" is a reference to key-value pairs) > > Current workaround at least for consistency: > l = list() # new empty list > t = tuple() # new empty tuple > s = set() # new empty set > d = dict() # new empty dictionary > > However, it doesn't feel right to not be able to initialize an empty set as cleanly and > consistently as lists, tuples and dictionaries in both forms. This is a topic better fit to python-ideas. -- ~Ethan~

1 0

Proposal to associate thread-local data/context with a faulthandler traceback
by tom＠tomforb.es 11 Mar '22

11 Mar '22

The faulthandler module is invaluable for tracking down segfaults in native code, however it is really lacking the ability to add some kind of useful breadcumb to aide debugging. Imagine you are running a large-scale distributed job over tens of millions of images and a single one causes opencv to segfault reliably. This can be very difficult to track down. It would be very useful to be able to add a string (limited by size) via the faulthandler module that is outputted along with the traceback. For example: ``` for image in millions_of_images(): faulthandler.context(f'{image.id=}') segfaulty_function(image) # Or maybe: for image in millions_of_images(): with faulthandler.context(f'{image.id=}'): segfaulty_function(image) ``` The traceback might be something like: ``` Fatal Python error: Segmentation fault Current thread 0x00007fb899f39700 (most recent call first): .... (traceback) Segmentation fault Context: image_id=foo ``` You could of course add logging to this function to print out the image ID before you run it through `segfaulty_function`, but if you're running this at high volume on a huge number of machines, this becomes a bit of an overhead, and it doesn't handle multiple threads running the function well.

4 6

suprocess.check_call and friends include STDERR in error message
by mangelozzi＠gmail.com 09 Mar '22

09 Mar '22

I often wish to perform the action of `subprocess.check_output`, i.e. print STDOUT live, and check the return code is 0 on exit. However if there is an error, I would like the STDERR to be printed. When looking at log files and just seeing the return code is often not very beneficial. E.g.: ```py subprocess.check_call(['ping', '-c', '1', 'github.com2']) ``` Will generate this error: ``` subprocess.CalledProcessError: Command '['ping', '-c', '1', 'github.com2']' returned non-zero exit status 2. ``` Here we are left wondering why it failed, on real world examples it becomes critical to be able to see what the actual error is. So I find I always have to wrap such calls in something like this to be able to see STDERR: ``` import subprocess cmds = ['ping', '-c', '1', 'github.com2'] result = subprocess.run(cmds, stderr=subprocess.PIPE) if result.returncode != 0: msg = result.stderr.decode().strip() raise subprocess.CalledProcessError(f"CALLED SUBPROCESS ERROR: Command: {' '.join(cmds)}\nReturn code {result.returncode}\nSTDERR: {msg}\n") ``` With the above one gets to see that STDERR is: ``` ping: github.com2: Temporary failure in name resolution ```

2 1

shutil.copyfileobj to return number of bytes copied
by Davis, Matthew 07 Mar '22

07 Mar '22

Hi, Currently shutil.copyfileobj returns nothing. I would like to be able to find out how many bytes were copied. Whilst most file-like objects have a .tell() which you could use, some don’t, and .tell() is not guaranteed to measure the number of bytes, it could measure with other units. I don’t think changing the return value from None to not-None would be backwards incompatible. We could also do the same change for all other shutil.copy* methods (except copytree). Looking at the code, this seems straightforward to implement. Existing code: ``` def copyfileobj(fsrc, fdst, length=0): """copy data from file-like object fsrc to file-like object fdst""" # Localize variable access to minimize overhead. if not length: length = COPY_BUFSIZE fsrc_read = fsrc.read fdst_write = fdst.write while True: buf = fsrc_read(length) if not buf: break fdst_write(buf) ``` New code: ``` def copyfileobj(fsrc, fdst, length=0): """copy data from file-like object fsrc to file-like object fdst""" # Localize variable access to minimize overhead. if not length: length = COPY_BUFSIZE fsrc_read = fsrc.read fdst_write = fdst.write bytes_copied = 0 while True: buf = fsrc_read(length) if not buf: break fdst_write(buf) bytes_copied += len(buf) return bytes_copied ``` Regards, Matt Matthew Davis Data Analytics Senior Lead Telstra Energy – Decision Science & AI E: Matthew.Davis.2(a)team.telstra.com<mailto:Matthew.Davis.2@team.telstra.com> | M: 0415762868

3 3

Add function for reading a specific number of bytes/characters from a file object that fails noisily
by Kevin Mills 05 Mar '22

05 Mar '22

The code I'm currently working on involves parsing binary data. If I ask for, say, 4 bytes, it's because I actually need 4 bytes and if the file doesn't have 4 bytes for me, it's malformed. Because `f.read(4)` can silently return less than 4 bytes and I don't want to have to explicitly double check every read, I'm using a wrapper function. def read_exact(f, size): data = f.read(size) if len(data) < size: raise EOFError(f"expected read of size {size}, got {len(data)}") return data I don't think my scenario of "give me exactly the number of bytes/characters I asked for or fail noisily" is particularly uncommon, so I think that a similar function should be added to the standard library somewhere. I guess as a function in `io`? Admittedly, as my own code demonstrates, implementing it yourself if you need it is trivial, so it may not actually be worth adding.

5 8

repeat until
by lynneandallan＠optusnet.com.au 02 Mar '22

02 Mar '22

Has anyone considered the idea of adding a "do at least once" loop to Python? This is frequently referred to as a do ... while or repeat ... until. At the moment, it's a bit of a hack to achieve this in that we do a 'while True: ( do thing ; if cond: ( break ) )'. Since I don't know how to format these messages, I've used '{' for line-beak-and-indent, ')' for line-break-and-dedent, and ';' for line-break-keeping-same-indent-level. My initial thoughts are that it would be reasonably easy to add a 'repeat: ( do thing ) until condition' which would far better specify intent of the loop (despite the possibility of break, while-true loops give no indication that it's not an infinite loop. And using repeat...until will ensure whoever had to add the code to the Python interpreter wouldn't have any clashes with the current while loop. Thoughts, anyone? Anyone? Bueller? :-)

10 19

Bringing the print statement back
by Guido van Rossum 02 Mar '22

02 Mar '22

In Python 3.10 we will no longer be burdened by the old parser (though 3rd party tooling needs to catch up). One thing that the PEG parser makes possible in about 20 lines of code is something not entirely different from the old print statement. I have a prototype: Python 3.10.0a0 (heads/print-statement-dirty:5ed19fcc1a, Jun 9 2020, 16:31:17) [Clang 11.0.0 (clang-1100.0.33.8)] on darwin Type "help", "copyright", "credits" or "license" for more information. Cannot read termcap database; using dumb terminal settings. >>> print 2+2 4 >>> print "hello world" hello world >>> print "hello", input("Name:") Name:Guido hello Guido >>> print 1, 2, 3, sep=", " 1, 2, 3 >>> But wait, there's more! The same syntax will make it possible to call *any* function: >>> len "abc" 3 >>> Or any method: >>> import sys >>> sys.getrefcount "abc" 24 >>> Really, *any* method: >>> class C: ... def foo(self, arg): print arg ... >>> C().foo 2+2 4 >>> There are downsides too, though. For example, you can't call a method without arguments: >>> print <built-in function print> >>> Worse, the first argument cannot start with a parenthesis or bracket: >>> print (1, 2, 3) 1 2 3 >>> C().foo (1, 2, 3) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: C.foo() takes 2 positional arguments but 4 were given >>> print (2+2), 42 4 (None, 42) >>> C().foo [0] Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'method' object is not subscriptable >>> No, it's not April 1st. I am seriously proposing this (but I'll withdraw it if the response is a resounding "boo, hiss"). After all, we currently have a bunch of complexity in the parser just to give a helpful error message to people used to Python 2's print statement: >>> print 1, 2, 3 File "<stdin>", line 1 print 1, 2, 3 ^ SyntaxError: Missing parentheses in call to 'print'. Did you mean print(1, 2, 3)? >>> And IIRC there have been a number of aborted attempts at syntactic hacks to allow people to call functions (like print) without parentheses, although (I think) none of them made it into a PEP. The PEG parser makes this much simpler, because it can simply backtrack -- by placing the grammar rule for this syntax (tentatively called "call statement") last in the list of alternatives for "small statement" we ensure that everything that's a valid expression statement (including print() calls) is still an expression statement with exactly the same meaning, while still allowing parameter-less function calls, without lexical hacks. (There is no code in my prototype that checks for a space after 'print' -- it just checks that there's a name, number or string following a name, which is never legal syntax.) One possible extension I didn't pursue (yet -- dare me!) is to allow parameter-less calls inside other expressions. For example, my prototype does not support things like this: >>> a = (len "abc") File "<stdin>", line 1 a = (len "abc") ^ SyntaxError: invalid syntax >>> I think that strikes a reasonable balance between usability and reduced detection of common errors. I could also dial it back a bit, e.g. maybe it's too much to allow 'C().foo x' and we should only allow dotted names (sufficient to access functions in imported modules and method calls on variables). Or maybe we should only allow simple names (allowing 'len x' but disallowing 'sys.getrefcount x'. Or maybe we should really only bring back print statements. I believe there are some other languages that support a similar grammar (Ruby? R? Raku?) but I haven't investigated. Thoughts? -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…>

42 66

Re: repeat until
by Rob Cliffe 02 Mar '22

02 Mar '22

Without testing, I am sure it would be slower. I suppose it would be reasonable if exceptions are raised in multiple places and the exceptions were given meaningful names. Best wishes Rob Cliffe On 01/03/2022 17:22, om wrote: > How does `try/except` (with raise AppropriateException inside the > block) compare to a len-1 loop? > > Om > > > > ---- On Tue, 01 Mar 2022 10:04:31 -0600 *python-ideas(a)python.org * > wrote ---- > > I have use cases for "do exactly once". > Basically a sequence of actions which can be broken off (when > something goes wrong and the whole process should be aborted, or > when something succeeds and there is no need to try alternatives) > at various points with `break`. Thus avoiding multiple if...then > indentation levels. > Of course it can be spelled as > for _ in '1': > for _ in ['once']: > etc. etc., so this is only a cosmetic need. > Best wishes > Rob Cliffe > > On 01/03/2022 15:19, Eric Fahlgren wrote: > > As you probably suspect, yes, it comes up every couple of > years. Here's one of the recent threads (there are more, just > search for 'until' in the archives), that might give you some > ideas for how this discussion will progress. :) > > https://mail.python.org/archives/list/python-ideas@python.org/thread/EDNARF… > <https://mail.python.org/archives/list/python-ideas@python.org/thread/EDNARF…> > > On Tue, Mar 1, 2022 at 7:09 AM <lynneandallan(a)optusnet.com.au > <mailto:lynneandallan@optusnet.com.au>> wrote: > > Has anyone considered the idea of adding a "do at least > once" loop to Python? This is frequently referred to as a > do ... while or repeat ... until. > > At the moment, it's a bit of a hack to achieve this in > that we do a 'while True: ( do thing ; if cond: ( break ) > )'. Since I don't know how to format these messages, I've > used '{' for line-beak-and-indent, ')' for > line-break-and-dedent, and ';' for > line-break-keeping-same-indent-level. > > My initial thoughts are that it would be reasonably easy > to add a 'repeat: ( do thing ) until condition' which > would far better specify intent of the loop (despite the > possibility of break, while-true loops give no indication > that it's not an infinite loop. > > And using repeat...until will ensure whoever had to add > the code to the Python interpreter wouldn't have any > clashes with the current while loop. > > Thoughts, anyone? Anyone? Bueller? :-) > _______________________________________________ > Python-ideas mailing list -- python-ideas(a)python.org > <mailto:python-ideas@python.org> > To unsubscribe send an email to > python-ideas-leave(a)python.org > <mailto:python-ideas-leave@python.org> > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > <https://mail.python.org/mailman3/lists/python-ideas.python.org/> > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/2PM6Q… > <https://mail.python.org/archives/list/python-ideas@python.org/message/2PM6Q…> > Code of Conduct: http://python.org/psf/codeofconduct/ > <http://python.org/psf/codeofconduct/> > > > _______________________________________________ > Python-ideas mailing list --python-ideas(a)python.org <mailto:python-ideas@python.org> > To unsubscribe send an email topython-ideas-leave(a)python.org <mailto:python-ideas-leave@python.org> > https://mail.python.org/mailman3/lists/python-ideas.python.org/ <https://mail.python.org/mailman3/lists/python-ideas.python.org/> > Message archived athttps://mail.python.org/archives/list/python-ideas@python.org/message/BYN… <https://mail.python.org/archives/list/python-ideas@python.org/message/BYNJ6…> > Code of Conduct:http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/> > > > _______________________________________________ > Python-ideas mailing list -- python-ideas(a)python.org > To unsubscribe send an email to python-ideas-leave(a)python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/QAGHJ… > Code of Conduct: http://python.org/psf/codeofconduct/ > >

3 3