Mailman 3 August 2016 - Python-ideas

add a hash to .pyc to don't mess between .py and .pyc
by Xavier Combelle Aug. 15, 2016

Aug. 15, 2016

I have stumbled upon several time with the following problem. I delete a module and the .pyc stay around. and by "magic", python still use the .pyc A similar error happen (but less often) when by some file system manipulation the .pyc happen to be newer than the .py but correspond to an older version of .py. It is not a major problem but it is still an existing problem. I'm not the first one to have this problem. A stack overflow search lead to quite a lot of relevant answers http://stackoverflow.com/search?q=old+pyc and google search too https://www.google.fr/search?q=old+pyc moreover several result of google result in bug tracking of various project. (There is also in these result the fact that .pyc are stored in VCS repositories but this is another problem not related) I even found a blog post using .pyc as a backdoor http://secureallthethings.blogspot.fr/2015/11/backdooring-python-via-pyc-pi… My idea to kill both bird in one stone would be to add a hash (likely to be cryptographic) of the .py file in the .pyc file and read the .py file and check the hash The additional cost of first startup cost will be just the hash calculation which I think is cheap comparing to other factors (especially input output) The additional second startup cost of a program the main cost will be the additional read of .py files and the cheap hash calculations. I believe the removing of the bugs would worth the performance cost. I know that some use case makes a use of just using .pyc and not keeping .py around, for example by not distribute the source file. But in my vision, this uses case should be solved per opt-in decision and not as a default. Several opt-in mechanisms could be envisioned: environment variables, command line switches, special compilation of .pyc which explicitly ask to not check for the hash. -- Xavier

6 8

Re: [Python-ideas] From mailing list to GitHub issues
by Arek Bulski Aug. 15, 2016

Aug. 15, 2016

Praise the guide! (Guido) GitHub issues are also delivered by email, with full post content. Guido and others will be satisfied. And mailing lists also send you messages in whatever freakin interface they provide it. And on my android gmail app it aint pretty. Most of it is auto replies in plain text and replaying to a particular thread is kinda impossible. I had to bring my laptop to reply to this. As Donald pointed out, there are people who are not going to create custom email processing toolchains. I am one of them. Moderation is not of concern to me particularly but that is also an advantage.

14 18

Fix default encodings on Windows
by Steve Dower Aug. 14, 2016

Aug. 14, 2016

I suspect there's a lot of discussion to be had around this topic, so I want to get it started. There are some fairly drastic ideas here and I need help figuring out whether the impact outweighs the value. Some background: within the Windows API, the preferred encoding is UTF-16. This is a 16-bit format that is typed as wchar_t in the APIs that use it. These APIs are generally referred to as the *W APIs (because they have a W suffix). There are also (broadly deprecated) APIs that use an 8-bit format (char), where the encoding is assumed to be "the user's active code page". These are *A APIs. AFAIK, there are no cases where a *A API should be preferred over a *W API, and many newer APIs are *W only. In general, Python passes byte strings into the *A APIs and text strings into the *W APIs. Right now, sys.getfilesystemencoding() on Windows returns "mbcs", which translates to "the system's active code page". As this encoding generally cannot represent all paths on Windows, it is deprecated and Unicode strings are recommended instead. This, however, means you need to write significantly different code between POSIX (use bytes) and Windows (use text). ISTM that changing sys.getfilesystemencoding() on Windows to "utf-8" and updating path_converter() (Python/posixmodule.c; likely similar code in other places) to decode incoming byte strings would allow us to undeprecate byte strings and add the requirement that they *must* be encoded with sys.getfilesystemencoding(). I assume that this would allow cross-platform code to handle paths similarly by encoding to whatever the sys module says they should and using bytes consistently (starting this thread is meant to validate/refute my assumption). (Yes, I know that people on POSIX should just change to using Unicode and surrogateescape. Unfortunately, rather than doing that they complain about Windows and drop support for the platform. If you want to keep hitting them with the stick, go ahead, but I'm inclined to think the carrot is more valuable here.) Similarly, locale.getpreferredencoding() on Windows returns a legacy value - the user's active code page - which should generally not be used for any reason. The one exception is as a default encoding for opening files when no other information is available (e.g. a Unicode BOM or explicit encoding argument). BOMs are very common on Windows, since the default assumption is nearly always a bad idea. Making open()'s default encoding detect a BOM before falling back to locale.getpreferredencoding() would resolve many issues, but I'm also inclined towards making the fallback utf-8, leaving locale.getpreferredencoding() solely as a way to get the active system codepage (with suitable warnings about it only being useful for back-compat). This would match the behavior that the .NET Framework has used for many years - effectively, utf_8_sig on read and utf_8 on write. Finally, the encoding of stdin, stdout and stderr are currently (correctly) inferred from the encoding of the console window that Python is attached to. However, this is typically a codepage that is different from the system codepage (i.e. it's not mbcs) and is almost certainly not Unicode. If users are starting Python from a console, they can use "chcp 65001" first to switch to UTF-8, and then *most* functionality works (input() has some issues, but those can be fixed with a slight rewrite and possibly breaking readline hooks). It is also possible for Python to change the current console encoding to be UTF-8 on initialize and change it back on finalize. (This would leave the console in an unexpected state if Python segfaults, but console encoding is probably the least of anyone's worries at that point.) So I'm proposing actively changing the current console to be Unicode while Python is running, and hence sys.std[in|out|err] will default to utf-8. So that's a broad range of changes, and I have little hope of figuring out all the possible issues, back-compat risks, and flow-on effects on my own. Please let me know (either on-list or off-list) how a change like this would affect your projects, either positively or negatively, and whether you have any specific experience with these changes/fixes and think they should be approached differently. To summarise the proposals (remembering that these would only affect Python 3.6 on Windows): * change sys.getfilesystemencoding() to return 'utf-8' * automatically decode byte paths assuming they are utf-8 * remove the deprecation warning on byte paths * make the default open() encoding check for a BOM or else use utf-8 * [ALTERNATIVE] make the default open() encoding check for a BOM or else use sys.getpreferredencoding() * force the console encoding to UTF-8 on initialize and revert on finalize So what are your concerns? Suggestions? Thanks, Steve

12 50

Consider adding clip or clamp function to math
by Neil Girdhar Aug. 13, 2016

Aug. 13, 2016

It's common to want to clip (or clamp) a number to a range. This feature is commonly needed for both floating point numbers and integers: http://stackoverflow.com/questions/9775731/clamping-floating-numbers-in-pyt… http://stackoverflow.com/questions/4092528/how-to-clamp-an-integer-to-some-… There are a few approaches: * use a couple ternary operators (e.g. https://github.com/scipy/scipy/pull/5944/files line 98, which generated a lot of discussion) * use a min/max construction, * call sorted on a list of the three numbers and pick out the first, or * use numpy.clip. Am I right that there is no *obvious* way to do this? If so, I suggest adding math.clip (or math.clamp) to the standard library that has the meaning: def clip(number, lower, upper): return lower if number < lower else upper if number > upper else number This would work for non-numeric types so long as the non-numeric types support comparison. It might also be worth adding assert lower < upper to catch some bugs. Best, Neil

19 60

Re: [Python-ideas] Fix default encodings on Windows
by Adam Bartoš Aug. 13, 2016

Aug. 13, 2016

Stephen J. Turnbull writes: > The exception is the proposed console changes, because there you *do* > perform all I/O with OS APIs. But I don't know anything about the > Windows console except that nobody seems happy with it. > > I'm quite happy with it. I mean, it's far from perfect, and when you look at discussions on Stack Overflow regarding Unicode on Windows console, almost everyone blames Windows console, but I think that software often doesn't communicate with it correctly. When Windows Unicode wide APIs are used, it just works. Adam Bartoš

1 0

Re: [Python-ideas] Fix default encodings on Windows
by Adam Bartoš Aug. 13, 2016

Aug. 13, 2016

On Fri Aug 12 19:03:38 EDT 2016 Victor Stinner wrote: > For the Windows console: I played with all Windows functions, tried all > fonts and many code pages. I also read technical blog articles of Microsoft > employees. I gave up on this issue. It doesn't seem possible to support > fully Unicode the Windows console (at least the last time I checked). By > the way, it seems like Windows functions have bugs, and the code page 65001 > fixes a few issues but introduces new issues... > > Do you mean that it doesn't seem possible to support Unicode on the Windows console by means of ANSI codepages? Because using the wide APIs seems to work (as win_unicode_console shows). There are some issues like non-BMP characters, which are encoded as surrogate pairs and the console doesn't understand them for display (shows two boxes), but this is just matter of display and not of corruption of the actual data (e.g. you can copy the text from console). Also, there seems to be no font to support all Unicode and AFAIK you cannot configure the console to use multiple fonts, but again this is a display issue of the console window itself rather than of the essential communication between Python and console. Adam Bartoš

1 0

Re: [Python-ideas] Fix default encodings on Windows
by Adam Bartoš Aug. 12, 2016

Aug. 12, 2016

*On Fri Aug 12 11:33:35 EDT 2016, * *Random832 wrote:*> On Wed, Aug 10, 2016, at 15:08, Steve Dower wrote: >>* That's the hope, though that module approaches the solution differently *>>* and may still uses. An alternative way for us to fix this whole thing *>>* would be to bring win_unicode_console into the standard library and use *>>* it by default (or probably whenever PYTHONIOENCODING is not specified). *> > I have concerns about win_unicode_console: > - For the "text_transcoded" streams, stdout.encoding is utf-8. For the > "text" streams, it is utf-16. UTF-16 it the "native" encoding since it corresponds to the wide chars used by Read/WriteConsoleW. The UTF-8 is used just as a signal for the consumers of PyOS_Readline. > - There is no object, as far as I can find, which can be used as an > unbuffered unicode I/O object. There is no buffer just on those wrapping streams because the bytes I have are not in UTF-8. Adding one would mean a fake buffer that just decodes and writes to the text stream. AFAIK there is no guarantee that sys.std* objects have buffer attribute and any code relying on that is incorrect. But I inderstand that there may be such code and we may want to be compatible. > - raw output streams silently drop the last byte if an odd number of > bytes are written. That's not true, it doesn't write an odd number of bytes, but returns the correct number of bytes written. If only one byte is given, it raises a ValueError. > - The sys.stdout obtained via streams.enable does not support .buffer / > .buffer.raw / .detach > - All of these objects provide a fileno() interface. Is this wrong? If I remember, I provide it because of some check -- maybe in input() -- to be viewed as a stdio stream. > - When using os.read/write for data that represents text, the data still > should be encoded in the console encoding and not in utf-8 or utf-16. I don't know what to do with this. Generally I wouldn't use bytes to communicate textual data. Regards, Adam Bartoš

2 1

Re: [Python-ideas] Fix default encodings on Windows
by Adam Bartoš Aug. 12, 2016

Aug. 12, 2016

Eryk Sun wrote: > IMO, Python needs a C implementation of the win_unicode_console > module, using the wide-character APIs ReadConsoleW and WriteConsoleW. > Note that this sets sys.std*.encoding as UTF-8 and transcodes, so > Python code never has to work directly with UTF-16 encoded text. > > The transcoding wrappers with 'utf-8' encoding are used just as a work around the fact that Python tokenizer cannot use utf-16-le and that the readlinehook machinery is unfortunately bytes-based. The tanscoding wrapper just has encoding 'utf-8' and no buffer attribute, so there is no actual transcoding in sys.std* objects. It's just a signal for PyOS_Readline consumers, and the transcoding occurs in a custom readline hook. Nothing like this would be needed if PyOS_Readline was replaced by some Python API wrapper around sys.readlinehook that would be Unicode string based. Adam Bartoš

2 1

Re: [Python-ideas] Fix default encodings on Windows
by Adam Bartoš Aug. 11, 2016

Aug. 11, 2016

> > On 11 August 2016 at 04:10, Steve Dower <steve.dower at python.org <https://mail.python.org/mailman/listinfo/python-ideas>> wrote: > >>* I suspect there's a lot of discussion to be had around this topic, so I want to get it started. There are some fairly drastic ideas here and I need help figuring out whether the impact outweighs the value. > * > My main reaction would be that if Drekin (Adam Bartoš) agrees the > changes natively solve the problems thathttps://pypi.python.org/pypi/win_unicode_console works around, it's > probably a good idea. > > The status quo is also sufficiently broken from both a native Windows > perspective and a cross-platform compatibility perspective that your > proposals are highly unlikely to make things *worse* :) > > Cheers, > Nick. > > The main idea of win_unicode_console is simple: to use WinAPI functions ReadConsoleW and WriteConsoleW to communicate with the interactive console on Windows and to wrap this in standard Python IO hierarchy – that's why sys.std*.encoding would be 'utf-16-le': it corresponds to widechar strings used by Windows wide APIs. But this is only about sys.std*.encoding, which I think is not so imporant. AFAIK sys.std*.encoding should be used only when you want to communicate in bytes (which I think is not a good idea), so it tells you, which encoding is sys.std*.buffer assuming. In fact sys.std* may even not have the buffer attribute, so its encoding attribute would be useless in that case. Unfortunatelly, sys.std*.encoding is used in some other places – namely by the consumers of the old PyOS_Readline API (the tokenizer and input) use it to decode the bytes returned. Actually, the consumers assume differente encodings (sys.stdin.encoding vs. sys.stdout.encoding), so it is impossible to write a correct readline hook when the encodings are not the same. So I think it would be nice to have Python and string-based implementation of readline hooks – sys.readlinehook attribute, which would use sys.std* by default on Windows and GNU readline on Unix. Nevertheless, I think it is a good idea to have more 'utf-8' defaults (or 'utf-8-readsig' for open()). I don't know whether it helps with the console issue to open the standard streams in 'utf-8'. Adam Bartoš

1 0

Re: [Python-ideas] Trial balloon: adding variable type declarations in support of PEP 484
by אלעזר Aug. 11, 2016

Aug. 11, 2016

> > class Starship: > stats: class Dict[str, int] = {} # Pure class variable > damage: class int = 0 # Hybrid class/instance variable > captain: str # Pure instance variable > I can't avoid noting that there is an opportunity here to insert NamedTuple into the core language. The above example is almost there, except it's mutable and without convenient methods. But class Starship(tuple): damage: int = 0 captain: str = "Kirk" Is an obvious syntax for Starship = NamedTuple('Starship', [('damage', int), ('captain', str)]) Only much more available and intuitive to read, use, and of course - type check. (Of course, it does mean adding semantics to the declaration syntax in general) I'm not really suggesting to make this change now, but I believe it will be done, sooner or later. My brief experience with mypy convinced me that it must be the case. The new declaration syntax only makes it easier. ~Elazar

5 16