Python-Dev March 2015

python-dev@python.org

90 participants
68 discussions

Re: [Python-Dev] Thoughts on running Python 3.5 on Windows (path, pip install --user, etc)

by Steve Dower

Paul Moore wrote: > I just thought I'd give installing Python 3.5 a go on my PC, now that > 3.5a2 has come out. I didn't get very far (see earlier message) but it prompted > me to think about how I'd use it, and what habits I'd need to change. > > I'd suggest that the "what's new in 3.5" document probably needs a section on > the new installer that explains this stuff... This is true. Right now I'm in experimentation mode, and being more aggressive about changing things than is probably a good idea (because it solicits feedback like this :) ). When things settle down I expect to end up closer to where we started, so there's not a huge amount of value in writing it all up right now. I'll get there. > First of all, I always use "all users" installs, so I have Python in "Program > Files" now. As a result, doing "pip install foo" requires elevation. As that's a > PITA, I probably need to switch to using "pip install --user". All that's fine, > and from there "py -3.5" works fine, as does "py -3.5 -m foo". But even if it > is, not every entry point has a corresponding "-m" invocation (pygments' > pygmentize command doesn't seem to, for example) I know you're already involved in this Paul, but for everyone else there's a big discussion going on at https://github.com/pypa/pip/issues/1668 about changing pip's default behaviour to handle falling back to --user automatically. > But suppose I want to put Python 3.5 on my PATH. The installer has an "add > Python to PATH" checkbox, but (if I'm reading the WiX source right, I didn't > select that on install) that doesn't add the user directory. So I need to add > that to my PATH. Is that right? And of course, that means I need to *know* the > user site directory ($env:LOCALAPPDATA\Python\Python35\Scripts), correct? Correct. There's no way to add a per-user directory to PATH from an all-users installation (except for a few approaches that I expect/hope would trigger malware detectors...) > Maybe the answer is that we simply start recommending that everyone on Windows > uses per-user installs. It makes little difference to me (beyond the fact that > when I want to look at the source of something in the stdlib, the location of > the file is a lot harder to remember than C:\Apps\Python34\Lib\whatever.py) but > I doubt it's what most people will expect. I'm okay with this. Installing for all users is really something that could be considered an advanced option rather than the default, especially since the aim (AIUI) of the all-users install is to pretend that Python was shipped with the OS. (I'd kind of like to take that further by splitting things more sensibly between Program Files, Common Files and System32, but there's very little gain from that and much MUCH pain as long as people are still expecting C:\PythonXY installs...) Cheers, Steve

5 years, 4 months

Windows installer - File associations in "per user" installs

by Paul Moore

On 9 March 2015 at 15:37, Steve Dower <Steve.Dower(a)microsoft.com> wrote: >> Maybe the answer is that we simply start recommending that everyone on Windows >> uses per-user installs. It makes little difference to me (beyond the fact that >> when I want to look at the source of something in the stdlib, the location of >> the file is a lot harder to remember than C:\Apps\Python34\Lib\whatever.py) but >> I doubt it's what most people will expect. > > I'm okay with this. Installing for all users is really something that could be considered an advanced option rather than the default, especially since the aim (AIUI) of the all-users install is to pretend that Python was shipped with the OS. (I'd kind of like to take that further by splitting things more sensibly between Program Files, Common Files and System32, but there's very little gain from that and much MUCH pain as long as people are still expecting C:\PythonXY installs...) I've just tried a per-user install of Python 3.5a2. The machine in question previously had (and still has) a system install of 3.4, with "Make this Python the default" selected (so the .py extension is associated with that version and specifically the 3.4 launcher). I didn't get the option to associate .py files with 3.5 (there's *no way* I'd consider that to be advanced usage - if I'm installing Python, why wouldn't I want to associate it with .py files [1]) and I still seem to have .py associated with the 3.4 launcher, not the 3.5 one that's in my %APPDATA% folder. >cmd /c assoc .py .py=Python.File >cmd /c ftype python.file python.file="C:\WINDOWS\py.exe" "%1" %* I'm happy if a per-user install of 3.5 makes a per-user filetype association (assuming such a thing is possible, I've never tried it before) but it's absolutely not OK if we're planning on recommending an install type that doesn't create the right association. Paul [1] Given that I have 3.4 and am installing an experimental 3.5 version, it's not actually at all clear cut which version I want as my default. In all honesty, I don't think this decision is actually something that should be defaulted. Maybe the "don't make the user make any choices in the default selection" approach has gone a little too far here?

5 years, 4 months

PEP 471 Final: os.scandir() merged into Python 3.5

by Victor Stinner

Hi, FYI I commited the implementation of os.scandir() written by Ben Hoyt. I hope that it will be part of Python 3.5 alpha 2 (Ben just sent the final patch today). Please test this new feature. You may benchmark here. http://bugs.python.org/issue22524 contains some benchmark tools and benchmark results of older versions of the patch. The implementation was tested on Windows and Linux. I'm now watching for buildbots to see how other platforms like os.scandir(). Bad news: OpenIndiana doesn't support d_type: the dirent structure has no d_type field. I already fixed the implementation to support this case. os.scandir() is still useful on OpenIndiana, because the stat result is cached in a DirEntry, so only one syscall is required, instead of multiple, when multiple DirEntry methods are called (ex: entry.is_dir() and not entry.is_symlink()). Victor

5 years, 5 months

Re: [Python-Dev] boxing and unboxing data types

by Serhiy Storchaka

понеділок, 09-бер-2015 10:18:50 ви написали: > On Mon, Mar 9, 2015 at 10:10 AM, Serhiy Storchaka <storchaka(a)gmail.com> wrote: > > понеділок, 09-бер-2015 09:52:01 ви написали: > > > On Mon, Mar 9, 2015 at 2:07 AM, Serhiy Storchaka <storchaka(a)gmail.com> > > > > And to be ideal drop-in replacement IntEnum should override such methods > > > > as __eq__ and __hash__ (so it could be used as mapping key). If all methods > > > > should be overridden to quack as int, why not take an int? > > > > > > You're absolutely right that if *all the methods should be overrriden to > > > quack as int, then you should subclass int (the Liskov substitution > > > principle). But all methods should not be overridden — mainly the methods > > > you overrode in your patch should be exposed. Here is a list of methods on > > > int that should not be on IntFlags in my opinion (give or take a couple): > > > > > > __abs__, __add__, __delattr__, __divmod__, __float__, __floor__, > > > __floordiv__, __index__, __lshift__, __mod__, __mul__, __pos__, __pow__, > > > __radd__, __rdivmod__, __rfloordiv__, __rlshift__, __rmod__, __rmul__, > > > __round__, __rpow__, __rrshift__, __rshift__, __rsub__, __rtruediv__, > > > __sub__, __truediv__, __trunc__, conjugate, denominator, imag, numerator, > > > real. > > > > > > I don't think __index__ should be exposed either since are you really going > > > to slice a list using IntFlags? Really? > > > > Definitely __index__ should be exposed. __int__ is for lossy conversion to int > > (as in float). __index__ is for lossless conversion. > > Is it? __index__ promises lossless conversion, but __index__ is *for* > indexing. I spite of its name it is for any lossless conversion. > > __add__ should be exposed because some code can use + instead of | for > > combining flags. But it shouldn't preserve the type, because this is not > > recommended way. > > I think it should be blocked because it can lead to all kinds of weird > bugs. If the flag is already set and you add it a copy, it silently spills > over into other flags. This is a mistake that a good interface prevents. I think this is a case when backward compatibility has larger weight. > > For the same reason I think __lshift__, __rshift__, __sub__, > > __mul__, __divmod__, __floordiv__, __mod__, etc should be exposed too. So the > > majority of the methods should be exposed, and there is a risk that we loss > > something. > > I totally disagree with all of those. > > > For good compatibility with Python code IntFlags should expose also > > __subclasscheck__ or __subclasshook__. And when we are at this point, why not > > use int subclass? > > Here's another reason. What if someone wants to use an IntFlags object, > but wants to use a fixed width type for storage, say numpy.int32? Why > shouldn't they be able to do that? By using composition, you can easily > provide such an option. You can design abstract interface Flags that can be combined with int or other type. But why you want to use numpy.int32 as storage? This doesn't save much memory, because with composition the IntFlags class weighs more than int subclass.

5 years, 5 months

Re: [Python-Dev] Tunning binary insertion sort algorithm in Timsort.

by nha pham

I do not know exactly, one thing I can imagine is: it turns the worst case of binary insertion sort to best case. With sorted array in range of 32 or 64 items, built from zero element. The new element you put into the sorted list has a high chance of being the smallest or the the highest of the sorted list (or nearly highest or nearly smallest) If that case happen, the old binary insertion sort will have the investigate all the list, while with my idea, it just have to compare more 1-2 times. I will try to run more test an more thinking to make sure though. On Mon, Mar 9, 2015 at 11:48 AM, nha pham <phqnha(a)gmail.com> wrote: > I do not know exactly, one thing I can imagine is: it turns the worst case > of binary insertion sort to best case. > With sorted array in range of 32 or 64 items, built from zero element. The > new element you put into the sorted list has a high chance of being the > smallest or the the highest of the sorted list (or nearly highest or nearly > smallest) > > If that case happen, the old binary insertion sort will have the > investigate all the list, while with my idea, it just have to compare more > 1-2 times. > I will try to run more test an more thinking to make sure though. > > > > On Mon, Mar 9, 2015 at 10:39 AM, Isaac Schwabacher <ischwabacher(a)wisc.edu> > wrote: > >> On 15-03-08, nha pham >> wrote: >> > >> > We can optimize the TimSort algorithm by optimizing its binary >> insertion sort. >> > >> > The current version of binary insertion sort use this idea: >> > >> > Use binary search to find a final position in sorted list for a new >> element X. Then insert X to that location. >> > >> > I suggest another idea: >> > >> > Use binary search to find a final postion in sorted list for a new >> element X. Before insert X to that location, compare X with its next >> element. >> > >> > For the next element, we already know if it is lower or bigger than X, >> so we can reduce the search area to the left side or on the right side of X >> in the sorted list. >> >> I don't understand how this is an improvement, since with binary search >> the idea is that each comparison cuts the remaining list to search in half; >> i.e., each comparison yields one bit of information. Here, you're spending >> a comparison to cut the list to search at the element you just inserted, >> which is probably not right in the middle. If you miss the middle, you're >> getting on average less than a full bit of information from your >> comparison, so you're not reducing the remaining search space by as much as >> you would be if you just compared to the element in the middle of the list. >> >> > I have applied my idea on java.util. ComparableTimSort.sort() and >> testing. The execute time is reduced by 2%-6% with array of random integer. >> >> For all that, though, experiment trumps theory... >> >> > Here is detail about algorithm and testing: >> https://github.com/nhapq/Optimize_binary_insertion_sort >> > >> > Sincerely. >> > >> > phqnha >> > >

5 years, 5 months

Tunning binary insertion sort algorithm in Timsort.

by nha pham

We can optimize the TimSort algorithm by optimizing its binary insertion sort. The current version of binary insertion sort use this idea: Use binary search to find a final position in sorted list for a new element X. Then insert X to that location. I suggest another idea: Use binary search to find a final postion in sorted list for a new element X. Before insert X to that location, compare X with its next element. For the next element, we already know if it is lower or bigger than X, so we can reduce the search area to the left side or on the right side of X in the sorted list. I have applied my idea on java.util. ComparableTimSort.sort() and testing. The execute time is reduced by 2%-6% with array of random integer. Here is detail about algorithm and testing: https://github.com/nhapq/Optimize_binary_insertion_sort Sincerely. phqnha

5 years, 5 months

boxing and unboxing data types

by Ethan Furman

When data is passed from Python to a native library (such as in an O/S call), how does the unboxing of data types occur? For a specific instance, os.open allows an integer whose various bits express desired behavior as `flags` -- if flags is 1, for example, the file is open write-only. If I pass an int-like object to os.open, __int__ is called; if I pass a thin wrapper over int to os.open, __int__ is not called. So the real question: anywhere in Python where an int is expected (for lower-level API work), but not directly received, should __int__ (or __index__) be called? and failure to do so is a bug? Here's my simple test code: class Wrap: def __init__(self, value): self.value = value def __int__(self): print('__int__') return self.value def __index__(self): print('__index__') return self.value class Thin(int): def __int__(self): print('__int__') return super().__int__() def __index__(self): print('__index__') return super().__index__() two = Wrap(2) [0, 1, 2][two] # __index__ # 2 import struct struct.pack('i', two) # __index__ # b'\x02\x00\x00\x00' t = Thin(1) huh = os.open('blah.txt', t) # Traceback (most recent call last): # File "<stdin>", line 1, in <module> # FileNotFoundError: [Errno 2] No such file or directory: 'blah.txt' -- ~Ethan~

5 years, 5 months

Thoughts on running Python 3.5 on Windows (path, pip install --user, etc)

by Paul Moore

I just thought I'd give installing Python 3.5 a go on my PC, now that 3.5a2 has come out. I didn't get very far (see earlier message) but it prompted me to think about how I'd use it, and what habits I'd need to change. I'd suggest that the "what's new in 3.5" document probably needs a section on the new installer that explains this stuff... First of all, I always use "all users" installs, so I have Python in "Program Files" now. As a result, doing "pip install foo" requires elevation. As that's a PITA, I probably need to switch to using "pip install --user". All that's fine, and from there "py -3.5" works fine, as does "py -3.5 -m foo". But even if it is, not every entry point has a corresponding "-m" invocation (pygments' pygmentize command doesn't seem to, for example) But suppose I want to put Python 3.5 on my PATH. The installer has an "add Python to PATH" checkbox, but (if I'm reading the WiX source right, I didn't select that on install) that doesn't add the user directory. So I need to add that to my PATH. Is that right? And of course, that means I need to *know* the user site directory ($env:LOCALAPPDATA\Python\Python35\Scripts), correct? It feels to me like this might be a frustrating step backwards for Windows users who have recently (with the arrival of ensurepip) got to the point where they can just run Python with "Add to path" and then simply do pip install pygments pygmentize --help Maybe the answer is that we simply start recommending that everyone on Windows uses per-user installs. It makes little difference to me (beyond the fact that when I want to look at the source of something in the stdlib, the location of the file is a lot harder to remember than C:\Apps\Python34\Lib\whatever.py) but I doubt it's what most people will expect. I'm completely OK with the idea that we move to "Program Files" as a default location. And I have no real issue with Steve's position that the "Add to Path" option has issues that can't really be solved because of the way Windows constructs the PATH. But I know there have been a lot of people frustrated by the complicated instructions needed to get something like pygmentize working, for whom getting pip in 2.7 and 3.4 was a major improvement. So I think we need *some* documentation helping them deal with what could well seem like a step backwards in Python 3.5... If I knew what the best (recommended) answer was, I'd be happy to write it up. But I'm not sure I do (after all, the target audience is people for whom "Add C:\PythonXY\Scripts to your PATH" was a problem in the pre-3.4 days). Should I raise this as a bug against the 3.5 documentation? If so, should it be a release blocker for the final release? Paul

5 years, 5 months

[RELEASED] Python 3.5.0a2 is now available

by Larry Hastings

On behalf of the Python development community and the Python 3.5 release team, I'm thrilled to announce the availability of Python 3.5.0a2. Python 3.5.0a2 is the second alpha release of Python 3.5, which will be the next major release of Python. Python 3.5 is still under heavy development, and is far from complete. This is a preview release, and its use is not recommended for production settings. Two important notes for Windows users about Python 3.5.0a2: * If you have previously installed Python 3.5.0a1, you must manually uninstall it before installing Python 3.5.0a2 (issue23612). * If installing Python 3.5.0a2 as a non-privileged user, you may need to escalate to administrator privileges to install an update to your C runtime libraries. You can find Python 3.5.0a2 here: https://www.python.org/downloads/release/python-350a2/ Happy hacking, //arry/

5 years, 5 months

Asking for review for Windows issues 21518 and 22080

by Claudiu Popa

Hello, The winreg module has a function for loading a registry key under another registry key, called winreg.LoadKey. Unfortunately, the module doesn't provide a way to unload that key after the user finishes operating with it. There's a patch [1] for exporting the RegUnloadKey [2] API in winreg module as winreg.UnloadKey, similar to how RegLoadKey is exported as winreg.LoadKey. The patch is helped by another one [3], which provides a new module, test.support.windows_helper, for handling various issues on the Windows platform, such as acquiring or releasing a privilege. Unfortunately, it seems there's a dearth of reviewers for this platform. Could someone knowledgeable with Windows be so kind to review these patches? They could make a good addition for Python 3.5. Thank you very much. [1] http://bugs.python.org/issue21518 - Expose RegUnloadKey in winreg [2] https://msdn.microsoft.com/en-us/library/windows/desktop/ms724924%28v=vs.85… [3] http://bugs.python.org/issue22080 - Add windows_helper module helper /Claudiu

5 years, 5 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

Python-Dev March 2015