Mailman 3 September 2015 - Python-Dev

[RELEASED] Python 3.5.0rc3 is now available
by Larry Hastings 08 Sep '15

08 Sep '15

On behalf of the Python development community and the Python 3.5 release team, I'm relieved to announce the availability of Python 3.5.0rc3, also known as Python 3.5.0 Release Candidate 3. The next release of Python 3.5 will be Python 3.5.0 final. There should be few (or no) changes to Python 3.5 between rc3 and final. This is a preview release, and its use is not recommended for production settings. You can find Python 3.5.0rc3 here: https://www.python.org/downloads/release/python-350rc3/ Windows and Mac users: please read the important platform-specific "Notes on this release" section near the end of that page. Happy hacking, //arry/

1 0

Critique of PEP 502 (String Interpolation Redux)
by Guido van Rossum 08 Sep '15

08 Sep '15

The text of the PEP has too much on motivation and rationale: maybe that would be suitable for an informative PEP. The proposal itself is under-specified. But the real weakness cannot be fixed by improving the text: it is in the key characteristic of the proposal, which wants to have its cake and eat it, too. This is initially likable: to recipients who don't know about e-strings they behave just like regular strings; but recipients who *do* know about them can extract the template and the values to be interpolated and use these to build an alternative string, e.g. to be used as a translation. However, in order to fulfill the former requirement, the type must inherit from str (it's one of those built-in types for which duck typing doesn't always work). And at the same time this is a real problem, because it's too easy to lose the e-string. For example, any function that operates on strings will operate on e-strings, but the result will in general be a plain str instead of an e-string, which can be confusing for functions that want the e-string. But *not* inheriting from str causes the reverse problem. IOW, e-strings don't make a good alternative for f-strings -- they would have to be added as a second string type. And the proliferation of similar but different template string types cannot make anyone happy. -- --Guido van Rossum (python.org/~guido)

2 3

PyParallel update
by Trent Nelson 06 Sep '15

06 Sep '15

[CC'ing python-dev@ for those that are curious; please drop and keep follow-up discussion to python-ideas@] Hi folks, I've made a lot of progress on PyParallel since the PyCon dev summit (https://speakerdeck.com/trent/pyparallel-pycon-2015-language-summit); I fixed the outstanding breakage with generators, exceptions and whatnot. I got the "instantaneous Wiki search server" working[1] and implemented the entire TechEmpower Frameworks Benchmark Suite[2], including a PyParallel-friendly pyodbc module, allowing database connections and querying in parallel. [1]: https://github.com/pyparallel/pyparallel/blob/branches/3.3-px/examples/wiki… [2]: https://github.com/pyparallel/pyparallel/blob/branches/3.3-px/examples/tefb… I set up a landing page for the project: http://pyparallel.org And there was some good discussion on reddit earlier this week: https://www.reddit.com/r/programming/comments/3jhv80/pyparallel_an_experime… I've put together some documentation on the project, its aims, and the key parts of the solution regarding the parallelism through simple client/server paradigms. This documentation is available directly on the github landing page for the project: https://github.com/pyparallel/pyparallel Writing that documentation forced me to formalize (or at least commit) to the restrictions/trade-offs that PyParallel would introduce, and I'm pretty happy I was basically able to boil it down into a single rule: Don't persist parallel objects. That keeps the mental model very simple. You don't need to worry about locking or ownership or races or anything like that. Just don't persist parallel objects, that's the only thing you have to remember. It's actually really easy to convert existing C code or Python code into something that is suitable for calling from within a parallel callback by just ensuring that rule isn't violated. It took about four hours to figure out how NumPy allocated stuff and add in the necessary PyParallel-aware tweaks, and not that much longer for pyodbc. (Most stuff "just works", though.) (The ABI changes would mean this is a Python 4.x type of thing; there are fancy ways we could avoid ABI changes and get this working on Python 3.x, but, eh, I like the 4.x target. It's realistic.) The other thing that clicked is that asyncio and PyParallel would actually work really well together for exploiting client-driven parallelism (PyParallel really is only suited to server-oriented parallelism at the moment, i.e. serving HTTP requests in parallel). With asyncio, though, you could keep the main-thread/single-thread client-drives-computation paradigm, but have it actually dispatch work to parallel.server() objects behind the scenes. For example, in order to process all files in a directory in parallel, asyncio would request a directory listing (i.e. issue a GET /) which the PyParallel HTTP server would return, it would then create non-blocking client connections to the same server and invoke whatever HTTP method is desired to do the file processing. You can either choose to write the new results from within the parallel context (which could then be accessed as normal files via HTTP), or you could have PyParallel return json/bytes, which could then be aggregated by asyncio. Everything is within the same process, so you get all the benefits that provides (free access to anything within scope, like large data structures, from within parallel contexts). You can synchronously call back into the main thread from a parallel thread, too, if you wanted to update a complex data structure directly. The other interesting thing that documentation highlights is the advantage of the split brain "main thread vs parallel thread" GC and non-GC allocators. I'm not sure if I've ever extolled the virtue of such an approach on paper or in e-mail. It's pretty neat though and allows us to avoid a whole raft of problems that need to be solved when you have a single GC/memory model. Next steps: once 3.5 is tagged, I'm going to bite the bullet and rebase. That'll require a bit of churn, so if there's enough interest from others, I figured we'd use the opportunity to at least get it building again on POSIX (Linux/OSX/FreeBSD). From there people can start implementing the missing bits for implementing the parallel machinery behind the scenes. The parallel interpreter thread changes I made are platform agnostic, the implementation just happens to be on Windows at the moment; don't let the Windows-only thing detract from what's actually being pitched: a (working, demonstrably-performant) solution to "Python's GIL problem". Regards, Trent.

1 0

PEPs and PEP 8 changes
by Jim J. Jewett 06 Sep '15

06 Sep '15

PEP 498 is only the latest PEP where part of the concern is fear that it may encourage certain types of bad code. Would it be reasonable to ask PEPs to start including a section on any recommended changes to PEP8? (e.g., "If an embedded expression doesn't fit on a single line, factor it out to a named variable.") I realize that there will be times when best practices (or common mistakes) aren't obvious in advance, but I'm a bit uncomfortable with "PEP 8 will probably grow advice"... if we expect to need such advice, then we should probably include it from the start. (PEP 8 is, after all, only advice.) -jJ

3 2

PEP 501 Shell Command Examples
by Nikolaus Rath 06 Sep '15

06 Sep '15

Hi Nick, You are giving runcommand(sh(i"cat {filename}")) as an example that avoids injection attacks. While this is true, I think this is still a terrible anti-pattern[1] that should not be entombed in a PEP as a positive example. Could you consider removing it? (It doubly wastes resources by pointlessly calling a shell, and then by parsing & quoting the argument only for the shell to do the same in reverse). Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.«

2 2

Critique of PEP 501 (General purpose string interpolation)
by Guido van Rossum 05 Sep '15

05 Sep '15

I think it's too much effort for too little gain. The motivation feels very weak; surely writing os.system("echo " + message_from_user) is just as easy (as is the %s spelling), so the security issue can hardly be blamed on PEP 498. I also don't think that the current way to address such security issues is a big deal: - The subprocess module is complex for other reasons, and a simpler wrapper could easily be made; - Database wrappers have forever included their own solution for safely quoting query parameters, and people who still don't use that are not likely to care about i-strings either. - Logging: again, it's hard to beat the existing solution, which mostly comes down to using %r instead of %s for any user-supplied or otherwise unverified data. - HTML quoting is an art and I'm skeptical that the proposal will even work for that use case. -- --Guido van Rossum (python.org/~guido)

2 2

Should we use getentropy() for os.urandom()?
by haypo s 05 Sep '15

05 Sep '15

Hi, I followed discussions on the new systems getrandom() on Linux and getentropy() on OpenBSD. I wanted to use them in Python to avoid the need of a file descriptor to read /dev/urandom. Linux getrandom() is also more secure than /dev/urandom because it blocks until /dev/urandom is feeded with enough entropy. getentropy() and getrandom() are now used in Python 2.7.10, Python 3.4 and newer. Today, an issue was reported on Solaris because os.urandom() is much slower with Python 2.7.10: https://bugs.python.org/issue25003 It looks like Solaris has getrandom() and getentropy(), and getentropy() is slow. Now I'm not sure that I understood the purpose of getentropy() even on OpenBSD. Should it be used to feed a PRNG in user-space, or can it be used as a PRNG? http://www.openbsd.org/cgi-bin/man.cgi/OpenBSD-current/man2/getentropy.2?qu… If getentropy() must only be used to feed a PRNG (which only use a few bytes), we should remove the code using getentropy() (but getrandom() should be kept). Note: I didn't know that other operating systems supported getrandom() and getentropy()! The feature looks recent in Solaris: " Solaris 11.3 adds two new system calls, getrandom(2) and getentropy(2), for getting random bit streams or raw entropy." https://blogs.oracle.com/darren/entry/solaris_new_system_calls_getentropy (article published at July, 2015) Note2: There is an open discussion proposing to "Use arc4random under OpenBSD for os.urandom() if /dev/urandom is not present" https://bugs.python.org/issue22542 getentropy() issue in Python (closed): https://bugs.python.org/issue22585 getrandom() issue in Python (closed): https://bugs.python.org/issue22181 Victor

3 2

Summary of Python tracker Issues
by Python tracker 05 Sep '15

05 Sep '15

ACTIVITY SUMMARY (2015-08-28 - 2015-09-04) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5057 (+15) closed 31704 (+33) total 36761 (+48) Open issues with patches: 2257 Issues opened (28) ================== #24954: No way to generate or parse timezone as produced by datetime.i http://bugs.python.org/issue24954 opened by gvanrossum #24955: webbrowser broken on Mac OS X when using the BROWSER variable http://bugs.python.org/issue24955 opened by sconseil #24956: Default value for an argument that is not in the choices list http://bugs.python.org/issue24956 opened by Sworddragon #24957: python3 -mpdb gets stuck in an unexitable infinite prompt loop http://bugs.python.org/issue24957 opened by shlomif #24959: unittest swallows part of stack trace when raising AssertionEr http://bugs.python.org/issue24959 opened by chris.jerdonek #24960: Can't use pip or easy_install with embeddable zip file. http://bugs.python.org/issue24960 opened by Oleg N #24964: Add tunnel CONNECT response headers to httplib / http.client http://bugs.python.org/issue24964 opened by Thomas #24965: Implement PEP 498: Literal String Formatting http://bugs.python.org/issue24965 opened by eric.smith #24966: shutil.get_terminal_size() throws ValueError if stdout is deta http://bugs.python.org/issue24966 opened by rsc1975 #24969: functools.lru_cache: a way to set decorator arguments at runti http://bugs.python.org/issue24969 opened by Marek Otahal #24970: Make distutils.Command an ABC http://bugs.python.org/issue24970 opened by Antony.Lee #24972: IDLE: revisit text highlighting for inactive windows on win32 http://bugs.python.org/issue24972 opened by markroseman #24973: CFLAGS for Visual Studio http://bugs.python.org/issue24973 opened by christopher.hogan #24977: shutil copy to non-existant directory http://bugs.python.org/issue24977 opened by TheOrangeOne #24978: Contributing to Python 2x and 3x Documentation. Translation to http://bugs.python.org/issue24978 opened by overr1de #24980: Allow for providing 'on_new_thread' callback to 'concurrent.fu http://bugs.python.org/issue24980 opened by harlowja #24981: Add a test which uses the ast module to compile the stdlib http://bugs.python.org/issue24981 opened by brett.cannon #24982: shutil.make_archive doesn't archive empty directories http://bugs.python.org/issue24982 opened by eugenek #24984: document AF_BLUETOOTH socket address formats http://bugs.python.org/issue24984 opened by Tim.Tisdall #24985: Python install test fails - OpenSSL - "dh key too small" http://bugs.python.org/issue24985 opened by nagle #24988: IDLE: debugger context menus not working on Mac http://bugs.python.org/issue24988 opened by markroseman #24990: Foreign language support in turtle module http://bugs.python.org/issue24990 opened by Al.Sweigart #24991: Define instance mutability explicitly on type objects http://bugs.python.org/issue24991 opened by ncoghlan #24995: better exception message when an unsupported object is passed http://bugs.python.org/issue24995 opened by yselivanov #24996: IDLE: debugger local/global vars should not be editable http://bugs.python.org/issue24996 opened by markroseman #24997: mock.call_args compares as equal and not equal http://bugs.python.org/issue24997 opened by akaptur #24999: Segfault in test_re.test_sre_character_class_literals() when P http://bugs.python.org/issue24999 opened by haypo #25000: _mock_call does not properly grab args and kwargs http://bugs.python.org/issue25000 opened by paetling Most recent 15 issues with no replies (15) ========================================== #24997: mock.call_args compares as equal and not equal http://bugs.python.org/issue24997 #24985: Python install test fails - OpenSSL - "dh key too small" http://bugs.python.org/issue24985 #24972: IDLE: revisit text highlighting for inactive windows on win32 http://bugs.python.org/issue24972 #24970: Make distutils.Command an ABC http://bugs.python.org/issue24970 #24960: Can't use pip or easy_install with embeddable zip file. http://bugs.python.org/issue24960 #24954: No way to generate or parse timezone as produced by datetime.i http://bugs.python.org/issue24954 #24942: Remove domain from ipaddress.reverse_pointer property and add http://bugs.python.org/issue24942 #24940: RotatingFileHandler uses tell in non-binary mode to determine http://bugs.python.org/issue24940 #24936: Idle: handle 'raise' properly when running with subprocess (2. http://bugs.python.org/issue24936 #24925: Allow doctest to find line number of __test__ strings if forma http://bugs.python.org/issue24925 #24924: _posixsubprocess.c: sysconf() might not be async-signal-safe http://bugs.python.org/issue24924 #24923: Append system paths in setup.py instead of prepending http://bugs.python.org/issue24923 #24922: assertWarnsRegex doesn't allow multiple warning messages http://bugs.python.org/issue24922 #24917: time_strftime() Buffer Over-read http://bugs.python.org/issue24917 #24916: In sysconfig, don't rely on sys.version format http://bugs.python.org/issue24916 Most recent 15 issues waiting for review (15) ============================================= #24988: IDLE: debugger context menus not working on Mac http://bugs.python.org/issue24988 #24984: document AF_BLUETOOTH socket address formats http://bugs.python.org/issue24984 #24980: Allow for providing 'on_new_thread' callback to 'concurrent.fu http://bugs.python.org/issue24980 #24973: CFLAGS for Visual Studio http://bugs.python.org/issue24973 #24965: Implement PEP 498: Literal String Formatting http://bugs.python.org/issue24965 #24955: webbrowser broken on Mac OS X when using the BROWSER variable http://bugs.python.org/issue24955 #24938: Measure if _warnings.c is still worth having http://bugs.python.org/issue24938 #24937: Multiple problems in getters & setters in capsulethunk.h http://bugs.python.org/issue24937 #24934: django_v2 benchmark not working in Python 3.6 http://bugs.python.org/issue24934 #24930: test_ssl broker was fixed http://bugs.python.org/issue24930 #24927: multiprocessing.Pool hangs forever on segfault http://bugs.python.org/issue24927 #24925: Allow doctest to find line number of __test__ strings if forma http://bugs.python.org/issue24925 #24923: Append system paths in setup.py instead of prepending http://bugs.python.org/issue24923 #24917: time_strftime() Buffer Over-read http://bugs.python.org/issue24917 #24916: In sysconfig, don't rely on sys.version format http://bugs.python.org/issue24916 Top 10 most discussed issues (10) ================================= #24912: The type of cached objects is mutable http://bugs.python.org/issue24912 43 msgs #23517: datetime.utcfromtimestamp rounds results incorrectly http://bugs.python.org/issue23517 23 msgs #7175: Define a standard location and API for configuration files http://bugs.python.org/issue7175 15 msgs #23406: interning and list comprehension leads to unexpected behavior http://bugs.python.org/issue23406 12 msgs #23630: support multiple hosts in create_server/start_server http://bugs.python.org/issue23630 12 msgs #17849: Missing size argument in readline() method for httplib's class http://bugs.python.org/issue17849 10 msgs #24966: shutil.get_terminal_size() throws ValueError if stdout is deta http://bugs.python.org/issue24966 10 msgs #22798: time.mktime doesn't update time.tzname http://bugs.python.org/issue22798 7 msgs #24984: document AF_BLUETOOTH socket address formats http://bugs.python.org/issue24984 7 msgs #24635: test_typing is flaky http://bugs.python.org/issue24635 6 msgs Issues closed (30) ================== #7739: time.strftime may hung while trying to open /etc/localtime but http://bugs.python.org/issue7739 closed by belopolsky #23332: datetime.isoformat() -> explicitly mark UTC string as such http://bugs.python.org/issue23332 closed by belopolsky #23375: test_py3kwarn fails on Windows http://bugs.python.org/issue23375 closed by haypo #23623: Python 3.5 docs need to clarify how to set PATH, etc http://bugs.python.org/issue23623 closed by paul.moore #24707: Assertion failed in pymonotonic_new http://bugs.python.org/issue24707 closed by haypo #24881: _pyio checks that `os.name == 'win32'` instead of 'nt' http://bugs.python.org/issue24881 closed by serhiy.storchaka #24891: race condition in initstdio() (python aborts running under noh http://bugs.python.org/issue24891 closed by haypo #24913: deque.index() overruns deque boundary http://bugs.python.org/issue24913 closed by larry #24931: _asdict breaks when inheriting from a namedtuple http://bugs.python.org/issue24931 closed by rhettinger #24952: stack_size([size]) is actually stack_size(size=0) http://bugs.python.org/issue24952 closed by martin.panter #24953: ICC and version should be in COMPILER string when built with I http://bugs.python.org/issue24953 closed by python-dev #24958: Segfault in REPL after pressing "r" and PgUp twice (on Mageia http://bugs.python.org/issue24958 closed by r.david.murray #24961: shell stdout broken after exiting interactive python prompt http://bugs.python.org/issue24961 closed by ned.deily #24962: Unnecessary space in using/mac http://bugs.python.org/issue24962 closed by python-dev #24963: ipaddress.IPv6Network doc typo http://bugs.python.org/issue24963 closed by python-dev #24967: add errors='ignore' to print kwargs http://bugs.python.org/issue24967 closed by brett.cannon #24968: Python 3 raises Unicode errors with the xxx.UTF-8 locale http://bugs.python.org/issue24968 closed by haypo #24971: os.environ.get() does not return the Environment Value in Linu http://bugs.python.org/issue24971 closed by r.david.murray #24974: ICC on Windows 8.1: _decimal fails to compile with default fp http://bugs.python.org/issue24974 closed by python-dev #24975: Python 3.5 can't compile AST involving PEP 448 unpacking http://bugs.python.org/issue24975 closed by yselivanov #24976: Arithmetic/precision bug when using floating-point numbers http://bugs.python.org/issue24976 closed by ezio.melotti #24979: Allow timezone offsets greater than 24 hours http://bugs.python.org/issue24979 closed by belopolsky #24983: Wrong AttributeError propagation http://bugs.python.org/issue24983 closed by martin.panter #24986: It should be possible to build successfully without external l http://bugs.python.org/issue24986 closed by python-dev #24987: subprocess.Popen with shell=True doesn't create socket http://bugs.python.org/issue24987 closed by asduj #24989: scan_eol() Buffer Over-read http://bugs.python.org/issue24989 closed by larry #24992: collections.OrderedDict constructor (odict_new) doesn't handle http://bugs.python.org/issue24992 closed by haypo #24993: The C function PyCodec_NameReplaceErrors doesn't handle PyCaps http://bugs.python.org/issue24993 closed by haypo #24994: Python 3.4.2 64-bit Installer error (Please insert the disk:) http://bugs.python.org/issue24994 closed by zach.ware #24998: docs: subprocess.Popen example has extra/invalid parameter http://bugs.python.org/issue24998 closed by r.david.murray

1 0

tp_finalize vs tp_del sematics
by Valentine Sinitsyn 04 Sep '15

04 Sep '15

Hi everybody, I'm trying to get sense of PEP-0442 [1]. Most of the looks clear, however I wasn't able to answer myself one simple question: why it wasn't possible to implement proposed CI disposal scheme on top of tp_del? Common sense suggests that tp_del and tp_finalize have different semantics. For instance, with tp_finalize there is a guarantee that the object will be in a safe state, as documented at [2]. However, tp_del is not documented, and I have only vague idea of its guarantees. Are there any? Thanks for the clarification. 1. https://www.python.org/dev/peps/pep-0442/ 2. https://docs.python.org/3/c-api/typeobj.html -- Best regards, Valentine Sinitsyn

3 11

Creating server-side clones on hg.python.org is broken right now
by Larry Hastings 04 Sep '15

04 Sep '15

Howdy. I went here: https://hg.python.org/cpython/ I clicked on "Server-side clone". I filled out the form. I clicked on the "clone" button. I was rewarded with this error message: error An error occurred while processing your request: no such method: rclone HALP PLS FX, //arry/

1 0