Mailman 3 July 2014 - Python-Dev

== on object tests identity in 3.x
by Andreas Maier July 7, 2014

July 7, 2014

Am 2014-07-07 23:31, schrieb Rob Cliffe: > > On 07/07/2014 22:11, Jan Kaliszewski wrote: >> [snip] >> >> IMHO, in Python context, "value" is a very vague term. Quite often >> we can read it as the very basic (but not the only one) notion of >> "what makes objects being equal or not" -- and then saying that >> "objects are compared by value" is a tautology. >> >> In other words, what object's "value" is -- is dependent on its >> nature: e.g. the value of a list is what are the values of its >> consecutive (indexed) items; the value of a set is based on values of >> all its elements without notion of order or repetition; the value of >> a number is a set of its abstract mathematical properties that >> determine what makes objects being equal, greater, lesser, how >> particular arithmetic operations work etc... >> >> I think, there is no universal notion of "the value of a Python >> object". The notion of identity seems to be most generic (every >> object has it, event if it does not have any other property) -- and >> that's why by default it is used to define the most basic feature of >> object's *value*, i.e. "what makes objects being equal or not" (== >> and !=). Another possibility would be to raise TypeError but, as >> Ethan Furman wrote, it would be impractical (e.g. >> key-type-heterogenic dicts or sets would be practically impossible to >> work with). On the other hand, the notion of sorting order (< > <= >> >=) is a much more specialized object property. > Quite so. > > x, y = object(), object() > print 'Equal:', ' '.join(attr for attr in dir(x) if > getattr(x,attr)==getattr(y,attr)) > print 'Unequal:', ' '.join(attr for attr in dir(x) if > getattr(x,attr)!=getattr(y,attr)) > > Equal: __class__ __doc__ __new__ __subclasshook__ > Unequal: __delattr__ __format__ __getattribute__ __hash__ __init__ > __reduce__ __reduce_ex__ __repr__ __setattr__ __sizeof__ __str__ > > Andreas, what attribute or combination of attributes do you think > should be the "values" of x and y? > Rob Cliffe > Whatever the object's type defines to be the value. Which requires the presence of an __eq__() or __ne__() implementation. I could even live with a default implementation on type object that ANDs the equality of all instance data attributes and class data attributes, but that is not possible because type object does not have a notion of such data attributes. Reverting to using the identity for the value of an instance of type object is somehow helpless. It may make existing code work, but it is not very logical. I could even argue it makes some logical code fail, because while it reliably detects that the same objects are equal, it fails to detect that different objects may also be equal (at least under a sensible definition of value equality). Having said all this: As a few people already wrote, we cannot change the implementation within a major release. So the real question is how we document it. I'll try to summarize in a separate posting. Andy

1 0

Tracker Stats
by Ezio Melotti July 7, 2014

July 7, 2014

Hi, I added a new "stats" page to the bug tracker: http://bugs.python.org/issue?@template=stats The page can be reached from the sidebar of the bug tracker: Summaries -> Stats The data are updated once a week, together with the Summary of Python tracker issues. Best Regards, Ezio Melotti

6 11

PEP 471: scandir(fd) and pathlib.Path(name, dir_fd=None)
by Victor Stinner July 7, 2014

July 7, 2014

Hi, IMO we must decide if scandir() must support or not file descriptor. It's an important decision which has an important impact on the API. To support scandir(fd), the minimum is to store dir_fd in DirEntry: dir_fd would be None for scandir(str). scandir(fd) must not close the file descriptor, it should be done by the caller. Handling the lifetime of the file descriptor is a difficult problem, it's better to let the user decide how to handle it. There is the problem of the limit of open file descriptors, usually 1024 but it can be lower. It *can* be an issue for very deep file hierarchy. If we choose to support scandir(fd), it's probably safer to not use scandir(fd) by default in os.walk() (use scandir(str) instead), wait until the feature is well tested, corner cases are well known, etc. The second step is to enhance pathlib.Path to support an optional file descriptor. Path already has methods on filenames like chmod(), exists(), rename(), etc. Example: fd = os.open(path, os.O_DIRECTORY) try: for entry in os.scandir(fd): # ... use entry to benefit of entry cache: is_dir(), lstat_result ... path = pathlib.Path(entry.name, dir_fd=entry.dir_fd) # ... use path which uses dir_fd ... finally: os.close(fd) Problem: if the path object is stored somewhere and use after the loop, Path methods will fail because dir_fd was closed. It's even worse if a new directory uses the same file descriptor :-/ (security issue, or at least tricky bugs!) Victor

5 7

Pending issues
by Mark Lawrence July 6, 2014

July 6, 2014

The following is a list of the 18 pending issues on the bug tracker. All have been in this state for at least one month so I'm assuming that they can be closed or they wouldn't have been set to pending in the first place. Can somebody take a look at them with a view to closing them or setting them back to open if needed. 16221 tokenize.untokenize() "compat" mode misses the encoding when using an iterator 15600 expose the finder details used by the FileFinder path hook 12588 test_capi.test_subinterps() failed on OpenBSD (powerpc) 7979 connect_ex returns 103 often 17668 re.split loses characters matching ungrouped parts of a pattern 11204 re module: strange behaviour of space inside {m, n} 14518 Add bcrypt $2a$ to crypt.py 15883 Add Py_errno to work around multiple CRT issue 19919 SSL: test_connect_ex_error fails with EWOULDBLOCK 20026 sqlite: handle correctly invalid isolation_level 18228 AIX locale parsing failure 1602742 itemconfigure returns incorrect text property of text items 19954 test_tk floating point exception on my gentoo box with tk 8.6.1 21084 IDLE can't deal with characters above the range (U+0000-U+FFFF) 20997 Wrong URL fragment identifier in search result 6895 locale._parse_localename fails when localename does not contain encoding information 1669539 Improve Windows os.path.join (ntpath.join) "smart" joining 21231 Issue a python 3 warning when old style classes are defined. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com

1 0

Summary of Python tracker Issues
by Python tracker July 4, 2014

July 4, 2014

ACTIVITY SUMMARY (2014-06-27 - 2014-07-04) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4603 (-40) closed 29086 (+82) total 33689 (+42) Open issues with patches: 2150 Issues opened (34) ================== #8631: subprocess.Popen.communicate(...) hangs on Windows http://bugs.python.org/issue8631 reopened by terry.reedy #20155: Regression test test_httpservers fails, hangs on Windows http://bugs.python.org/issue20155 reopened by r.david.murray #21876: os.rename(src,dst) does nothing when src and dst files are har http://bugs.python.org/issue21876 opened by Aaron.Swan #21877: External.bat and pcbuild of tkinter do not match. http://bugs.python.org/issue21877 opened by terry.reedy #21878: wsgi.simple_server's wsgi.input read/readline waits forever in http://bugs.python.org/issue21878 opened by rschoon #21879: str.format() gives poor diagnostic on placeholder mismatch http://bugs.python.org/issue21879 opened by roysmith #21880: IDLE: Ability to run 3rd party code checkers http://bugs.python.org/issue21880 opened by sahutd #21881: python cannot parse tcl value http://bugs.python.org/issue21881 opened by schwab #21882: turtledemo modules imported by test___all__ cause side effects http://bugs.python.org/issue21882 opened by ned.deily #21883: relpath: Provide better errors when mixing bytes and strings http://bugs.python.org/issue21883 opened by Matt.Bachmann #21885: shutil.copytree hangs (on copying root directory of a lxc cont http://bugs.python.org/issue21885 opened by krichter #21886: asyncio: Future.set_result() called on cancelled Future raises http://bugs.python.org/issue21886 opened by haypo #21888: plistlib.FMT_BINARY behavior doesn't send required dict parame http://bugs.python.org/issue21888 opened by n8henrie #21889: https://docs.python.org/2/library/multiprocessing.html#process http://bugs.python.org/issue21889 opened by krichter #21890: wsgiref.simple_server sends headers on empty bytes http://bugs.python.org/issue21890 opened by rschoon #21895: signal.pause() doesn't wake up on SIGCHLD in non-main thread http://bugs.python.org/issue21895 opened by bkabrda #21896: Unexpected ConnectionResetError in urllib.request against a va http://bugs.python.org/issue21896 opened by Tymoteusz.Paul #21897: frame.f_locals causes segfault on Python >=3.4.1 http://bugs.python.org/issue21897 opened by msmhrt #21898: .hgignore: Missing ignores for Eclipse/pydev http://bugs.python.org/issue21898 opened by andymaier #21899: Futures are not marked as completed http://bugs.python.org/issue21899 opened by Sebastian.Kreft.Deezer #21901: test_selectors.PollSelectorTestCase.test_above_fd_setsize repo http://bugs.python.org/issue21901 opened by r.david.murray #21902: Docstring of math.acosh, asinh, and atanh http://bugs.python.org/issue21902 opened by kdavies4 #21903: ctypes documentation MessageBoxA example produces error http://bugs.python.org/issue21903 opened by Dan.O'Donovan #21905: RuntimeError in pickle.whichmodule when sys.modules if mutate http://bugs.python.org/issue21905 opened by Olivier.Grisel #21906: Tools\Scripts\md5sum.py doesn't work in Python 3.x http://bugs.python.org/issue21906 opened by torrin #21907: Update Windows build batch scripts http://bugs.python.org/issue21907 opened by zach.ware #21909: PyLong_FromString drops const http://bugs.python.org/issue21909 opened by h.venev #21910: File protocol should document if writelines must handle genera http://bugs.python.org/issue21910 opened by JanKanis #21911: "IndexError: tuple index out of range" should include the requ http://bugs.python.org/issue21911 opened by cool-RR #21913: Possible deadlock in threading.Condition.wait() in Python 2.7. http://bugs.python.org/issue21913 opened by sangeeth #21914: Create unit tests for Turtle guionly http://bugs.python.org/issue21914 opened by Lita.Cho #21915: telnetlib.Telnet constructor does not match telnetlib.Telnet._ http://bugs.python.org/issue21915 opened by yaneurabeya #21916: Create unit tests for turtle textonly http://bugs.python.org/issue21916 opened by ingrid #21917: Python 2.7.7 Tests fail, and math is faulty http://bugs.python.org/issue21917 opened by repcsike Most recent 15 issues with no replies (15) ========================================== #21916: Create unit tests for turtle textonly http://bugs.python.org/issue21916 #21909: PyLong_FromString drops const http://bugs.python.org/issue21909 #21899: Futures are not marked as completed http://bugs.python.org/issue21899 #21898: .hgignore: Missing ignores for Eclipse/pydev http://bugs.python.org/issue21898 #21889: https://docs.python.org/2/library/multiprocessing.html#process http://bugs.python.org/issue21889 #21885: shutil.copytree hangs (on copying root directory of a lxc cont http://bugs.python.org/issue21885 #21874: test_strptime fails on rhel/centos/fedora systems http://bugs.python.org/issue21874 #21865: Improve invalid category exception for warnings.filterwarnings http://bugs.python.org/issue21865 #21859: Add Python implementation of FileIO http://bugs.python.org/issue21859 #21854: Fix cookielib in unicodeless build http://bugs.python.org/issue21854 #21853: Fix inspect in unicodeless build http://bugs.python.org/issue21853 #21852: Fix optparse in unicodeless build http://bugs.python.org/issue21852 #21851: Fix gettext in unicodeless build http://bugs.python.org/issue21851 #21850: Fix httplib and SimpleHTTPServer in unicodeless build http://bugs.python.org/issue21850 #21847: Fix xmlrpc in unicodeless build http://bugs.python.org/issue21847 Most recent 15 issues waiting for review (15) ============================================= #21916: Create unit tests for turtle textonly http://bugs.python.org/issue21916 #21914: Create unit tests for Turtle guionly http://bugs.python.org/issue21914 #21907: Update Windows build batch scripts http://bugs.python.org/issue21907 #21906: Tools\Scripts\md5sum.py doesn't work in Python 3.x http://bugs.python.org/issue21906 #21905: RuntimeError in pickle.whichmodule when sys.modules if mutate http://bugs.python.org/issue21905 #21903: ctypes documentation MessageBoxA example produces error http://bugs.python.org/issue21903 #21902: Docstring of math.acosh, asinh, and atanh http://bugs.python.org/issue21902 #21898: .hgignore: Missing ignores for Eclipse/pydev http://bugs.python.org/issue21898 #21897: frame.f_locals causes segfault on Python >=3.4.1 http://bugs.python.org/issue21897 #21890: wsgiref.simple_server sends headers on empty bytes http://bugs.python.org/issue21890 #21883: relpath: Provide better errors when mixing bytes and strings http://bugs.python.org/issue21883 #21880: IDLE: Ability to run 3rd party code checkers http://bugs.python.org/issue21880 #21868: Tbuffer in turtle allows negative size http://bugs.python.org/issue21868 #21865: Improve invalid category exception for warnings.filterwarnings http://bugs.python.org/issue21865 #21862: cProfile command-line should accept "-m module_name" as an alt http://bugs.python.org/issue21862 Top 10 most discussed issues (10) ================================= #21902: Docstring of math.acosh, asinh, and atanh http://bugs.python.org/issue21902 13 msgs #21911: "IndexError: tuple index out of range" should include the requ http://bugs.python.org/issue21911 11 msgs #12067: Doc: remove errors about mixed-type comparisons. http://bugs.python.org/issue12067 8 msgs #20155: Regression test test_httpservers fails, hangs on Windows http://bugs.python.org/issue20155 8 msgs #12750: datetime.strftime('%s') should respect tzinfo http://bugs.python.org/issue12750 7 msgs #21090: File read silently stops after EIO I/O error http://bugs.python.org/issue21090 7 msgs #12420: distutils tests fail if PATH is not defined http://bugs.python.org/issue12420 6 msgs #14050: Tutorial, list.sort() and items comparability http://bugs.python.org/issue14050 6 msgs #21882: turtledemo modules imported by test___all__ cause side effects http://bugs.python.org/issue21882 6 msgs #2571: can cmd.py's API/docs for the use of an alternate stdin be imp http://bugs.python.org/issue2571 5 msgs Issues closed (72) ================== #2057: difflib: add patch capability http://bugs.python.org/issue2057 closed by terry.reedy #4899: doctest should support fixtures http://bugs.python.org/issue4899 closed by terry.reedy #5207: extend strftime/strptime format for RFC3339 and RFC2822 http://bugs.python.org/issue5207 closed by belopolsky #5638: test_httpservers fails CGI tests if --enable-shared http://bugs.python.org/issue5638 closed by ned.deily #5862: multiprocessing 'using a remote manager' example errors and po http://bugs.python.org/issue5862 closed by berker.peksag #5930: Transient error in multiprocessing (test_number_of_objects) http://bugs.python.org/issue5930 closed by haypo #6692: asyncore kqueue support http://bugs.python.org/issue6692 closed by haypo #7506: multiprocessing.managers.BaseManager.__reduce__ references Bas http://bugs.python.org/issue7506 closed by berker.peksag #7885: test_distutils fails if Python built in separate directory http://bugs.python.org/issue7885 closed by ned.deily #9860: Building python outside of source directory fails http://bugs.python.org/issue9860 closed by belopolsky #10000: mark more tests as CPython specific http://bugs.python.org/issue10000 closed by rhettinger #10236: Sporadic failures of test_ssl http://bugs.python.org/issue10236 closed by ned.deily #10402: sporadic test_bsddb3 failures http://bugs.python.org/issue10402 closed by jcea #10445: _ast py3k : add lineno back to "args" node http://bugs.python.org/issue10445 closed by Claudiu.Popa #10941: imaplib: Internaldate2tuple produces wrong result if date is n http://bugs.python.org/issue10941 closed by r.david.murray #11273: asyncore creates selec (or poll) on every iteration http://bugs.python.org/issue11273 closed by haypo #11279: test_posix and lack of "id -G" support - less noise required? http://bugs.python.org/issue11279 closed by python-dev #11389: unittest: no way to control verbosity of doctests from cmd http://bugs.python.org/issue11389 closed by terry.reedy #11453: asyncore.file_wrapper should implement __del__ and call close http://bugs.python.org/issue11453 closed by haypo #11762: Ast doc: warning and version number http://bugs.python.org/issue11762 closed by berker.peksag #12401: unset PYTHON* environment variables when running tests http://bugs.python.org/issue12401 closed by haypo #12498: asyncore.dispatcher_with_send, disconnection problem + miss-co http://bugs.python.org/issue12498 closed by haypo #12814: Possible intermittent bug in test_array http://bugs.python.org/issue12814 closed by ned.deily #12842: Docs: first parameter of tp_richcompare() always has the corre http://bugs.python.org/issue12842 closed by asvetlov #12876: Make Test Error : ImportError: No module named _sha256 http://bugs.python.org/issue12876 closed by gregory.p.smith #13103: copy of an asyncore dispatcher causes infinite recursion http://bugs.python.org/issue13103 closed by haypo #13413: time.daylight incorrect behavior in linux glibc http://bugs.python.org/issue13413 closed by belopolsky #13689: fix CGI Web Applications with Python link in howto/urllib2 http://bugs.python.org/issue13689 closed by berker.peksag #13985: Menu.tk_popup : menu doesn't disapear when main window is ico http://bugs.python.org/issue13985 closed by ned.deily #14069: In extensions (?...) the lookbehind assertion cannot choose be http://bugs.python.org/issue14069 closed by ezio.melotti #14097: Improve the "introduction" page of the tutorial http://bugs.python.org/issue14097 closed by zach.ware #14235: test_cmd.py does not correctly call reload() http://bugs.python.org/issue14235 closed by berker.peksag #14709: http.client fails sending read()able Object http://bugs.python.org/issue14709 closed by ned.deily #15014: smtplib: add support for arbitrary auth methods http://bugs.python.org/issue15014 closed by r.david.murray #15549: openssl version in windows builds does not support renegotiati http://bugs.python.org/issue15549 closed by ned.deily #15750: test_localtime_daylight_false_dst_true raises OverflowError: m http://bugs.python.org/issue15750 closed by haypo #15870: PyType_FromSpec should take metaclass as an argument http://bugs.python.org/issue15870 closed by belopolsky #16188: Windows C Runtime Library Mismatch http://bugs.python.org/issue16188 closed by rlinscheer #16474: More code coverage for imp module http://bugs.python.org/issue16474 closed by berker.peksag #17399: test_multiprocessing hang on Windows, non-sockets http://bugs.python.org/issue17399 closed by terry.reedy #18258: Fix test discovery for test_codecmaps*.py http://bugs.python.org/issue18258 closed by zach.ware #18592: Idle: test SearchDialogBase.py http://bugs.python.org/issue18592 closed by terry.reedy #19024: Document asterisk (*), splat or star operator http://bugs.python.org/issue19024 closed by terry.reedy #19870: Backport Cookie fix to 2.7 (httponly / secure flag) http://bugs.python.org/issue19870 closed by berker.peksag #20218: Add methods to `pathlib.Path`: `write_text`, `read_text`, `wri http://bugs.python.org/issue20218 closed by cool-RR #20961: Fix usages of the note directive in the documentation http://bugs.python.org/issue20961 closed by berker.peksag #21046: Document formulas used in statistics http://bugs.python.org/issue21046 closed by ezio.melotti #21151: winreg.SetValueEx causes crash if value = None http://bugs.python.org/issue21151 closed by python-dev #21447: Intermittent asyncio.open_connection / futures.InvalidStateErr http://bugs.python.org/issue21447 closed by haypo #21582: use support.captured_stdx context managers - test_asyncore http://bugs.python.org/issue21582 closed by python-dev #21652: Python 2.7.7 regression in mimetypes module on Windows http://bugs.python.org/issue21652 closed by python-dev #21679: Prevent extraneous fstat during open() http://bugs.python.org/issue21679 closed by pitrou #21755: test_importlib.test_locks fails --without-threads http://bugs.python.org/issue21755 closed by berker.peksag #21778: PyBuffer_FillInfo() from 3.3 http://bugs.python.org/issue21778 closed by skrah #21780: make unicodedata module 64-bit safe http://bugs.python.org/issue21780 closed by python-dev #21781: make _ssl module 64-bit clean http://bugs.python.org/issue21781 closed by haypo #21811: Anticipate fixes to 3.x and 2.7 for OS X 10.10 Yosemite suppor http://bugs.python.org/issue21811 closed by ned.deily #21856: memoryview: test slice clamping http://bugs.python.org/issue21856 closed by terry.reedy #21857: assert that functions clearing the current exception are not c http://bugs.python.org/issue21857 closed by haypo #21863: Display module names of C functions in cProfile http://bugs.python.org/issue21863 closed by pitrou #21871: Python 2.7.7 regression in mimetypes read_windows_registry http://bugs.python.org/issue21871 closed by python-dev #21884: turtle regression of issue #21823: "uncaught exception" on "AM http://bugs.python.org/issue21884 closed by ned.deily #21887: Python3 can't detect Tcl/Tk 8.6.1 http://bugs.python.org/issue21887 closed by ned.deily #21891: sysmodule.c, #define terminated with semicolon. http://bugs.python.org/issue21891 closed by ned.deily #21892: hashtable.c not using PY_FORMAT_SIZE_T http://bugs.python.org/issue21892 closed by python-dev #21893: unicodeobject.c not using PY_FORMAT_SIZE_T http://bugs.python.org/issue21893 closed by haypo #21894: ImportError: cannot import name jit http://bugs.python.org/issue21894 closed by ned.deily #21900: .hgignore: Missing ignores for downloaded doc build tools http://bugs.python.org/issue21900 closed by r.david.murray #21904: Multiple closures accessing the same non-local variable always http://bugs.python.org/issue21904 closed by r.david.murray #21908: Grammatical error in 3.4 tutorial http://bugs.python.org/issue21908 closed by r.david.murray #21912: Deferred logging may use outdated references http://bugs.python.org/issue21912 closed by vinay.sajip #777588: asyncore/Windows: select() doesn't report errors for a non-blo http://bugs.python.org/issue777588 closed by haypo

1 0

[Python-checkins] Daily reference leaks (42917d774476): sum=9
by Brett Cannon July 4, 2014

July 4, 2014

Looks like there is an actual leak found by test_io. Any ideas on what may have introduced it? On Fri Jul 04 2014 at 5:01:02 AM, <solipsis(a)pitrou.net> wrote: > results for 42917d774476 on branch "default" > -------------------------------------------- > > test_functools leaked [0, 0, 3] memory blocks, sum=3 > test_io leaked [2, 2, 2] references, sum=6 > > > Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', > '3:3:/home/antoine/cpython/refleaks/reflogODkfML', '-x'] > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > https://mail.python.org/mailman/listinfo/python-checkins >

1 0

My summary of the scandir (PEP 471)
by Victor Stinner July 2, 2014

July 2, 2014

Hi, @Ben: it's time to update your PEP to complete it with this discussion! IMO DirEntry must be as simple as possible and portable: - os.scandir(str) - DirEntry.lstat_result object only available on Windows, same result than os.lstat() - DirEntry.fullname(): os.path.join(directory, DirEntry.name), where directory would be an hidden attribute of DirEntry Notes: - DirEntry.lstat_result is better than DirEntry.lstat() because it makes explicitly that lstat_result is only computed once. When I call DirEntry.lstat(), I expect to get the current status of the file, not the cached one. It's also hard to explain (document) that DirEntry.lstat() may or may call a system call. Don't do that, use DirEntry.lstat_result. - I don't think that we should support scandir(bytes). If you really want to support os.scandir(bytes), it must raise an error on Windows since bytes filename are already deprecated. It wouldn't make sense to add new function with a deprecated feature. Since we have the PEP 383 (surrogateescape), it's better to advice to use Unicode on all platforms. Almost all Python functions are able to encode back Unicode filename automatically. Use os.fsencode() to encode manually if needd. - We may not define a DirEntry.fullname() method: the directory name is usually well known. Ok, but every time that I use os.listdir(), I write os.path.join(directory, name) because in some cases I want the full path. Example: interesting = [] for name in os.listdir(path): fullpath = os.path.join(path, name) if os.path.isdir(fullpath): continue if ... test on the file ...: # i need the full path here, not the relative path # (ex: my own recursive "scandir"/"walk" function) interesting.append(fullpath) - It must not be possible to "refresh" a DirEntry object. Call os.stat(entry.fullname()) or pathlib.Path(entry.fullname()) to get fresh data. DirEntry is only computed once, that's all. It's well defined. - No Windows wildcard, you wrote that the feature has many corner cases, and it's only available on Windows. It's easy to combine scandir with fnmatch. Victor

9 19

[RELEASE] Python 2.7.8
by Benjamin Peterson July 2, 2014

July 2, 2014

Greetings, I have the distinct privilege of informing you that the latest release of the Python 2.7 series, 2.7.8, has been released and is available for download. 2.7.8 contains several important regression fixes and security changes: - The openssl version bundled in the Windows installer has been updated. - A regression in the mimetypes module on Windows has been fixed. [1] - A possible overflow in the buffer type has been fixed. [2] - A bug in the CGIHTTPServer module which allows arbitrary execution of code in the server root has been patched. [3] - A regression in the handling of UNC paths in os.path.join has been fixed. [4] Downloads of 2.7.8 are at https://www.python.org/download/releases/2.7.8/ The full changelog is located at http://hg.python.org/cpython/raw-file/v2.7.8/Misc/NEWS This is a production release. As always, please report bugs to http://bugs.python.org/ Till next time, Benjamin Peterson 2.7 Release Manager (on behalf of all of Python's contributors) [1] http://bugs.python.org/issue21652 [2] http://bugs.python.org/issue21831 [3] http://bugs.python.org/issue21766 [4] http://bugs.python.org/issue21672

1 0

Network Security Backport Status
by Alex Gaynor July 1, 2014

July 1, 2014

Hi all, I wanted to bring everyone up to speed on the status of PEP 466, what's been completed, and what's left to do. First the completed stuff: * hmac.compare_digest * hashlib.pbkdf2_hmac Are both backported, and I've added support to use them in Django, so users should start seeing these benefits just as soon as we get a Python release into their hands. Now the uncompleted stuff: * Persistent file descriptor for ``os.urandom`` * SSL module It's the SSL module that I'll spend the rest of this email talking about. Backporting the features from the Python3 version of this module has proven more difficult than I had expected. This is primarily because the stdlib took a maintenance strategy that was different from what most Python projects have done for their 2/3 support: multiple independent codebases. I've tried a few different strategies for the backport, none of which has worked: * Copying the ``ssl.py``, ``test_ssl.py``, and ``_ssl.c`` files from Python3 and trying to port all the code. * Coping just ``test_ssl.py`` and then copying individual chunks/functions as necessary to get stuff to pass. * Manually doing stuff. All of these proved to be a massive undertaking, and made it too easy to accidentally introduce breaking changes. I've come up with a new approach, which I believe is most likely to be successful, but I'll need help to implement it. The idea is to find the most recent commit which is a parent of both the ``2.7`` and ``default`` branches. Then take every single change to an ``ssl`` related file on the ``default`` branch, and attempt to replay it on the ``2.7`` branch. Require manual review on each commit to make sure it compiles, and to ensure it doesn't make any backwards incompatible changes. I think this provides the most iterative and guided approach to getting this done. I can do all the work of reviewing each commit, but I need some help from a mercurial expert to automate the cherry-picking/rebasing of every single commit. What do folks think? Does this approach make sense? Anyone willing to help with the mercurial scripting? Cheers, Alex

4 3

PEP 471 -- os.scandir() function -- a better and faster directory iterator
by Ben Hoyt July 1, 2014

July 1, 2014

Hi Python dev folks, I've written a PEP proposing a specific os.scandir() API for a directory iterator that returns the stat-like info from the OS, the main advantage of which is to speed up os.walk() and similar operations between 4-20x, depending on your OS and file system. Full details, background info, and context links are in the PEP, which Victor Stinner has uploaded at the following URL, and I've also copied inline below. http://legacy.python.org/dev/peps/pep-0471/ Would love feedback on the PEP, but also of course on the proposal itself. -Ben PEP: 471 Title: os.scandir() function -- a better and faster directory iterator Version: $Revision$ Last-Modified: $Date$ Author: Ben Hoyt <benhoyt(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30-May-2014 Python-Version: 3.5 Abstract ======== This PEP proposes including a new directory iteration function, ``os.scandir()``, in the standard library. This new function adds useful functionality and increases the speed of ``os.walk()`` by 2-10 times (depending on the platform and file system) by significantly reducing the number of times ``stat()`` needs to be called. Rationale ========= Python's built-in ``os.walk()`` is significantly slower than it needs to be, because -- in addition to calling ``os.listdir()`` on each directory -- it executes the system call ``os.stat()`` or ``GetFileAttributes()`` on each file to determine whether the entry is a directory or not. But the underlying system calls -- ``FindFirstFile`` / ``FindNextFile`` on Windows and ``readdir`` on Linux and OS X -- already tell you whether the files returned are directories or not, so no further system calls are needed. In short, you can reduce the number of system calls from approximately 2N to N, where N is the total number of files and directories in the tree. (And because directory trees are usually much wider than they are deep, it's often much better than this.) In practice, removing all those extra system calls makes ``os.walk()`` about **8-9 times as fast on Windows**, and about **2-3 times as fast on Linux and Mac OS X**. So we're not talking about micro- optimizations. See more `benchmarks`_. .. _`benchmarks`: https://github.com/benhoyt/scandir#benchmarks Somewhat relatedly, many people (see Python `Issue 11406`_) are also keen on a version of ``os.listdir()`` that yields filenames as it iterates instead of returning them as one big list. This improves memory efficiency for iterating very large directories. So as well as providing a ``scandir()`` iterator function for calling directly, Python's existing ``os.walk()`` function could be sped up a huge amount. .. _`Issue 11406`: http://bugs.python.org/issue11406 Implementation ============== The implementation of this proposal was written by Ben Hoyt (initial version) and Tim Golden (who helped a lot with the C extension module). It lives on GitHub at `benhoyt/scandir`_. .. _`benhoyt/scandir`: https://github.com/benhoyt/scandir Note that this module has been used and tested (see "Use in the wild" section in this PEP), so it's more than a proof-of-concept. However, it is marked as beta software and is not extensively battle-tested. It will need some cleanup and more thorough testing before going into the standard library, as well as integration into `posixmodule.c`. Specifics of proposal ===================== Specifically, this PEP proposes adding a single function to the ``os`` module in the standard library, ``scandir``, that takes a single, optional string as its argument:: scandir(path='.') -> generator of DirEntry objects Like ``listdir``, ``scandir`` calls the operating system's directory iteration system calls to get the names of the files in the ``path`` directory, but it's different from ``listdir`` in two ways: * Instead of bare filename strings, it returns lightweight ``DirEntry`` objects that hold the filename string and provide simple methods that allow access to the stat-like data the operating system returned. * It returns a generator instead of a list, so that ``scandir`` acts as a true iterator instead of returning the full list immediately. ``scandir()`` yields a ``DirEntry`` object for each file and directory in ``path``. Just like ``listdir``, the ``'.'`` and ``'..'`` pseudo-directories are skipped, and the entries are yielded in system-dependent order. Each ``DirEntry`` object has the following attributes and methods: * ``name``: the entry's filename, relative to ``path`` (corresponds to the return values of ``os.listdir``) * ``is_dir()``: like ``os.path.isdir()``, but requires no system calls on most systems (Linux, Windows, OS X) * ``is_file()``: like ``os.path.isfile()``, but requires no system calls on most systems (Linux, Windows, OS X) * ``is_symlink()``: like ``os.path.islink()``, but requires no system calls on most systems (Linux, Windows, OS X) * ``lstat()``: like ``os.lstat()``, but requires no system calls on Windows The ``DirEntry`` attribute and method names were chosen to be the same as those in the new ``pathlib`` module for consistency. Notes on caching ---------------- The ``DirEntry`` objects are relatively dumb -- the ``name`` attribute is obviously always cached, and the ``is_X`` and ``lstat`` methods cache their values (immediately on Windows via ``FindNextFile``, and on first use on Linux / OS X via a ``stat`` call) and never refetch from the system. For this reason, ``DirEntry`` objects are intended to be used and thrown away after iteration, not stored in long-lived data structured and the methods called again and again. If a user wants to do that (for example, for watching a file's size change), they'll need to call the regular ``os.lstat()`` or ``os.path.getsize()`` functions which force a new system call each time. Examples ======== Here's a good usage pattern for ``scandir``. This is in fact almost exactly how the scandir module's faster ``os.walk()`` implementation uses it:: dirs = [] non_dirs = [] for entry in scandir(path): if entry.is_dir(): dirs.append(entry) else: non_dirs.append(entry) The above ``os.walk()``-like code will be significantly using scandir on both Windows and Linux or OS X. Or, for getting the total size of files in a directory tree -- showing use of the ``DirEntry.lstat()`` method:: def get_tree_size(path): """Return total size of files in path and subdirs.""" size = 0 for entry in scandir(path): if entry.is_dir(): sub_path = os.path.join(path, entry.name) size += get_tree_size(sub_path) else: size += entry.lstat().st_size return size Note that ``get_tree_size()`` will get a huge speed boost on Windows, because no extra stat call are needed, but on Linux and OS X the size information is not returned by the directory iteration functions, so this function won't gain anything there. Support ======= The scandir module on GitHub has been forked and used quite a bit (see "Use in the wild" in this PEP), but there's also been a fair bit of direct support for a scandir-like function from core developers and others on the python-dev and python-ideas mailing lists. A sampling: * **Nick Coghlan**, a core Python developer: "I've had the local Red Hat release engineering team express their displeasure at having to stat every file in a network mounted directory tree for info that is present in the dirent structure, so a definite +1 to os.scandir from me, so long as it makes that info available." [`source1 <http://bugs.python.org/issue11406>`_] * **Tim Golden**, a core Python developer, supports scandir enough to have spent time refactoring and significantly improving scandir's C extension module. [`source2 <https://github.com/tjguk/scandir>`_] * **Christian Heimes**, a core Python developer: "+1 for something like yielddir()" [`source3 <https://mail.python.org/pipermail/python-ideas/2012-November/017772.html>`_] and "Indeed! I'd like to see the feature in 3.4 so I can remove my own hack from our code base." [`source4 <http://bugs.python.org/issue11406>`_] * **Gregory P. Smith**, a core Python developer: "As 3.4beta1 happens tonight, this isn't going to make 3.4 so i'm bumping this to 3.5. I really like the proposed design outlined above." [`source5 <http://bugs.python.org/issue11406>`_] * **Guido van Rossum** on the possibility of adding scandir to Python 3.5 (as it was too late for 3.4): "The ship has likewise sailed for adding scandir() (whether to os or pathlib). By all means experiment and get it ready for consideration for 3.5, but I don't want to add it to 3.4." [`source6 <https://mail.python.org/pipermail/python-dev/2013-November/130583.html>`_] Support for this PEP itself (meta-support?) was given by Nick Coghlan on python-dev: "A PEP reviewing all this for 3.5 and proposing a specific os.scandir API would be a good thing." [`source7 <https://mail.python.org/pipermail/python-dev/2013-November/130588.html>`_] Use in the wild =============== To date, ``scandir`` is definitely useful, but has been clearly marked "beta", so it's uncertain how much use of it there is in the wild. Ben Hoyt has had several reports from people using it. For example: * Chris F: "I am processing some pretty large directories and was half expecting to have to modify getdents. So thanks for saving me the effort." [via personal email] * bschollnick: "I wanted to let you know about this, since I am using Scandir as a building block for this code. Here's a good example of scandir making a radical performance improvement over os.listdir." [`source8 <https://github.com/benhoyt/scandir/issues/19>`_] * Avram L: "I'm testing our scandir for a project I'm working on. Seems pretty solid, so first thing, just want to say nice work!" [via personal email] Others have `requested a PyPI package`_ for it, which has been created. See `PyPI package`_. .. _`requested a PyPI package`: https://github.com/benhoyt/scandir/issues/12 .. _`PyPI package`: https://pypi.python.org/pypi/scandir GitHub stats don't mean too much, but scandir does have several watchers, issues, forks, etc. Here's the run-down as of the stats as of June 5, 2014: * Watchers: 17 * Stars: 48 * Forks: 15 * Issues: 2 open, 19 closed **However, the much larger point is this:**, if this PEP is accepted, ``os.walk()`` can easily be reimplemented using ``scandir`` rather than ``listdir`` and ``stat``, increasing the speed of ``os.walk()`` very significantly. There are thousands of developers, scripts, and production code that would benefit from this large speedup of ``os.walk()``. For example, on GitHub, there are almost as many uses of ``os.walk`` (194,000) as there are of ``os.mkdir`` (230,000). Open issues and optional things =============================== There are a few open issues or optional additions: Should scandir be in its own module? ------------------------------------ Should the function be included in the standard library in a new module, ``scandir.scandir()``, or just as ``os.scandir()`` as discussed? The preference of this PEP's author (Ben Hoyt) would be ``os.scandir()``, as it's just a single function. Should there be a way to access the full path? ---------------------------------------------- Should ``DirEntry``'s have a way to get the full path without using ``os.path.join(path, entry.name)``? This is a pretty common pattern, and it may be useful to add pathlib-like ``str(entry)`` functionality. This functionality has also been requested in `issue 13`_ on GitHub. .. _`issue 13`: https://github.com/benhoyt/scandir/issues/13 Should it expose Windows wildcard functionality? ------------------------------------------------ Should ``scandir()`` have a way of exposing the wildcard functionality in the Windows ``FindFirstFile`` / ``FindNextFile`` functions? The scandir module on GitHub exposes this as a ``windows_wildcard`` keyword argument, allowing Windows power users the option to pass a custom wildcard to ``FindFirstFile``, which may avoid the need to use ``fnmatch`` or similar on the resulting names. It is named the unwieldly ``windows_wildcard`` to remind you you're writing power- user, Windows-only code if you use it. This boils down to whether ``scandir`` should be about exposing all of the system's directory iteration features, or simply providing a fast, simple, cross-platform directory iteration API. This PEP's author votes for not including ``windows_wildcard`` in the standard library version, because even though it could be useful in rare cases (say the Windows Dropbox client?), it'd be too easy to use it just because you're a Windows developer, and create code that is not cross-platform. Possible improvements ===================== There are many possible improvements one could make to scandir, but here is a short list of some this PEP's author has in mind: * scandir could potentially be further sped up by calling ``readdir`` / ``FindNextFile`` say 50 times per ``Py_BEGIN_ALLOW_THREADS`` block so that it stays in the C extension module for longer, and may be somewhat faster as a result. This approach hasn't been tested, but was suggested by on Issue 11406 by Antoine Pitrou. [`source9 <http://bugs.python.org/msg130125>`_] Previous discussion =================== * `Original thread Ben Hoyt started on python-ideas`_ about speeding up ``os.walk()`` * Python `Issue 11406`_, which includes the original proposal for a scandir-like function * `Further thread Ben Hoyt started on python-dev`_ that refined the ``scandir()`` API, including Nick Coghlan's suggestion of scandir yielding ``DirEntry``-like objects * `Final thread Ben Hoyt started on python-dev`_ to discuss the interaction between scandir and the new ``pathlib`` module * `Question on StackOverflow`_ about why ``os.walk()`` is slow and pointers on how to fix it (this inspired the author of this PEP early on) * `BetterWalk`_, this PEP's author's previous attempt at this, on which the scandir code is based .. _`Original thread Ben Hoyt started on python-ideas`: https://mail.python.org/pipermail/python-ideas/2012-November/017770.html .. _`Further thread Ben Hoyt started on python-dev`: https://mail.python.org/pipermail/python-dev/2013-May/126119.html .. _`Final thread Ben Hoyt started on python-dev`: https://mail.python.org/pipermail/python-dev/2013-November/130572.html .. _`Question on StackOverflow`: http://stackoverflow.com/questions/2485719/very-quickly-getting-total-size-… .. _`BetterWalk`: https://github.com/benhoyt/betterwalk Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

22 58