Mailman 3 April 2016 - Python-Dev

Tag-based buildmaster (was: Most 3.x buildbots are green again ... )
by Zachary Ware April 13, 2016

April 13, 2016

(Cross-posting to python-buildbots, discussion is probably best continued there) On Wed, Apr 13, 2016 at 3:37 PM, Brett Cannon <brett(a)python.org> wrote: > On Wed, 13 Apr 2016 at 13:17 Zachary Ware <zachary.ware+pydev(a)gmail.com> > wrote: >> After receiving a suggestion from koobs several months ago, I've been >> intermittently thinking about completely redoing our buildmaster setup >> such that instead of a single builder per version on each slave, we … [View More]

1 0

ping on issue 18378: locale.getdefaultlocale() fails on recent Mac OS X
by Chris Barker April 12, 2016

April 12, 2016

Hi folks, There have been multiple reports of folks having failures on startup of matplotlib, which appears to be due to the most recent OS-X version setting the locale weirdly. This was identified last summer in this issue: http://bugs.python.org/issue18378 It looks like the issue was figured out, and even a patch contributed, but it stalled out before being applied. I have no idea if the patch is any good, but it would be great to get this fixed! -Thanks, -Chris -- Christopher … [View More]

1 0

Maybe, just maybe, pathlib doesn't belong.
by Alexander Walters April 12, 2016

April 12, 2016

In reviewing the ongoing arguments about how to make pathlib better, there have been circular arguments about if it is even broken, if it should support bytes, if there should be a path protocol that all functions that touch the filesystem should use, if that protocol should support bytes, how that protocol should be open or closed to allow third party modules to act as paths, etc., etc. If there is headway being made, I do not see it. I don't think we can come to an agreement that will … [View More]

13 31

Challenge: Please break this! (a.k.a restricted mode revisited)
by Jon Ribbens April 12, 2016

April 12, 2016

I've made another attempt at Python sandboxing, which does something which I've not seen tried before - using the 'ast' module to do static analysis of the untrusted code before it's executed, to prevent most of the sneaky tricks that have been used to break out of past attempts at sandboxes. In short, I'm turning Python's usual "gentleman's agreement" that you should not access names and attributes that are indicated as private by starting with an underscore into a rigidly enforced rule: try … [View More]

20 70

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()
by INADA Naoki April 12, 2016

April 12, 2016

Sorry, I've forgot to use "Reply All". On Tue, Apr 12, 2016 at 9:49 AM, INADA Naoki <songofacandy(a)gmail.com> wrote: > IHMO it's safer to get an encoding error rather than no error when you >> concatenate two byte strings encoded to two different encodings (mojibake). >> >> print(os.fspath(obj)) will more likely do what you expect if os.fspath() >> always return str. I mean that it will encode your filename to the encoding >> of the terminal which can … [View More]

2 1

thoughts on backporting __wrapped__ to 2.7?
by Robert Collins April 12, 2016

April 12, 2016

I'm working on teaching funcsigs - the backport of inspect.signature - better handling for wrapped functions, and the key enabler to do that is capturing the wrapped function in __wrapped__. I'm wondering what folks thoughts are on backporting that to 2.7 - seems cleaner than monkeypatching functools.wraps, which would tend to be subject to import ordering races and general ick. I'll likely prep such a monkeypatch for folk that are stuck on older versions of 2.7 anyhow... so its not a huge win..… [View More]

4 5

pathlib+os/shutil feedback
by Sven R. Kunze April 11, 2016

April 11, 2016

I talked to my colleague. He didn't remember the concrete use-case, though, he instantly mentioned three possible things (no order of preference): 1) pathlib + mtime 2) os.walk and pathlib 3) creation/removal of paths He wasn't too sure but I checked with the docs and his memories seemed to be correct: ----- 1) https://docs.python.org/3/library/pathlib.html#pathlib.Path.stat High-level path objects should return high-level [insert type here] objects. Put differently, an API for … [View More]retrieving time-stats as real date/time objects would be nice. I think that can be expanded to other pathlib methods as well, to make them less "os-wrapper"-like and provide added value. ----- 2) I remember a discussion on python-ideas about using "glob" or "rglob". However, when searching the docs for "walk" like in "os.walk" or for "iter", I don't find "glob"/"rglob". I can imagine ourselves (pathlib newbies back then), we didn't discover them. It would be great if the docs could be improved like the following: """ Path.rglob(pattern) Walk down a given path; a wrapper for "os.scandir"/"os.listdir". This is like calling glob() with “**” added in front of the given pattern: """ I think it would make "glob" and "rglob" more discoverable to new users. NOTE: """ Using the “**” pattern in large directory trees may consume an inordinate amount of time.""" sounds not really encouraging. This is especially true for "rglob" as it is defined as "like calling glob() with “**”". That leads to wondering whether "rglob" performs slow globbing instead of a "os.scandir"/"os.listdir". https://docs.python.org/3/library/pathlib.html#basic-use even promotes "glob" with "**" in the beginning which seems rather discouraging to use "rglob" as a fast alternative to "os.walk/scandir/listdir". Renaming "rglob"/adding a "scan" method would definitely help here. ----- 3) Again searching the docs for "create", "delete" (nothing found) and "remove", I found "Path.touch", "Path.rmdir" and "Path.unlink". It would be great if we had an easy way to remove a complete subtree as with "shutil.rmtree". We mostly don't care if a directory is empty. We need the system to be in a state of "this path does not exist anymore". Moreover, touching a file is good enough to "create" it if you don't care about changing its mtime. It you care about its mtime, it's a problem to "touch". ------ That's it for our issues with pathlib from the past. Additionally, I got two further observations: A) pathlib tries to mimic/publish some low-level APIs to its users. "unlink" is not something people would expect to use when they want to "delete" or to "remove" a file or a directory. I know where the term stems from but it's the wrong layer of abstraction IMHO. Same for "touch" or "chmod". B) "rename" vs "replace". The difference is not really clear from the docs. You need to read "Path.replace" in order to understand "Path.rename" completely. (one raises an exception, the other don't if I read it correctly). If there's some agreement to change things with respect to those 5 points, I am willing to put some time into it. Best, Sven [View Less]

3 3

Re: [Python-Dev] [Python-checkins] cpython (2.7): Issue #25910: Fixed more links in the docs.
by Tim Golden April 11, 2016

April 11, 2016

On 11/04/2016 15:38, serhiy.storchaka wrote: > - <http://www.openssl.org/docs/apps/ciphers.html#CIPHER_LIST_FORMAT>`__. > + <http://www.openssl.org/docs/apps/ciphers.html#CIPHER-LIST-FORMAT>`__. Is there any intended irony in our link to openssl not being via https? :) TJG

2 1

Pathlib enhancments - method name only
by Ethan Furman April 11, 2016

April 11, 2016

On 04/08/2016 10:46 AM, Koos Zevenhoven wrote: > On Fri, Apr 8, 2016 at 7:42 PM, Chris Barker wrote: >> On Fri, Apr 8, 2016 at 9:02 AM, Koos Zevenhoven wrote: >>> I'm still thinking a little bit about 'pathname', which to me sounds >>> more like a string than fspath does. >> >> >> I like that a lot - or even "__pathstr__" or "__pathstring__" >> after all, we're making a big deal out of the fact that a path is >> *not a string*, … [View More]

15 33

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)
by Anders Munch April 11, 2016

April 11, 2016

Steven D'Aprano: > although I must admit I don't understand why the entire OS is effected. A consequence of memory overcommit, I'd wager. The crasher code not only allocates vast swathes of memory, but accesses it as well, which is bad news for Linux with overcommit enabled. When the OS runs out of backing store to handle page faults, anything can happen. - Anders

1 0

tp_new selection regression in the 2.7 branch
by Julien Cristau April 11, 2016

April 11, 2016

Hi, changeset https://hg.python.org/cpython/rev/e7062dd9085e in the 2.7 branch changes how tp_new is assigned, and causes regressions with multiple inheritance from extension classes. http://bugs.python.org/issue25731#msg262922 has a fairly simple reproducer using cython. The __base__ attribute is set correctly, but tp_new is now wrong and thus the object initialization is broken. Can this change be fixed or reverted before the next 2.7.x release? (I have not verified if this regression … [View More]

1 0

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)
by Wes Turner April 10, 2016

April 10, 2016

On Apr 10, 2016 11:51 AM, "Jon Ribbens" <jon+python-dev(a)unequivocal.co.uk> wrote: > > On Sun, Apr 10, 2016 at 02:51:23PM +1000, Nick Coghlan wrote: > > On 9 April 2016 at 22:43, Victor Stinner <victor.stinner(a)gmail.com> wrote: > > > See pysandbox test suite for a lot of ways to escape a sandbox. CPython has > > > a list of know code to crash CPython (I don't recall the dieectory in > > > sources), even with the latest version of CPython. &… [View More]gt; > > > They're at https://hg.python.org/cpython/file/tip/Lib/test/crashers > > Thanks. I take your point that sandboxing Python requires CPython to > free of code execution bugs. However I will note that none of the > crashers in that directory will work inside my experiment (except > "infinite_loop_re.py", which isn't a crasher just a long loop). > > > Even without those considerations though, there are system level > > denial of service attacks that untrusted code can perform without even > > trying to break out of the sandbox - the most naive is "while 1: > > pass", but there are more interesting ones like "from itertools import > > count; sum(count())", or even "sum(iter(int, 1))" and "list(iter(int, > > 1))". > > Yes, of course. I have already explicitly noted that infinite loops > and memory exhausation are not preventable. > > > Operating system level security sandboxes still aren't particularly > > easy to use correctly, but they're a lot more reliable than language > > runtime level sandboxes, can be used to defend against many more > > attack vectors, and even offer increased flexibility (e.g. "can write > > to these directories, but no others", "can read these files, but no > > others", "can contact these IP addresses, but no others"). > > I don't entirely trust operating system sandboxes either - I generally > assume that if someone can execute arbitrary code on my machine, then > they can do anything they want to that machine. > > What I *might* trust, though, would be a "sandbox Python" that is > itself running inside an operating system sandbox... > * https://github.com/jupyter/jupyterhub/wiki/Spawners - Docker LXC Containers - https://github.com/jupyter/jupyterhub/wiki/Authenticators - DOS is still trivial - Segfault is still trivial * http://doc.pypy.org/en/latest/sandbox.html#introduction _______________________________________________ > Python-Dev mailing list > Python-Dev(a)python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com [View Less]

1 0

When should pathlib stop being provisional?
by Brett Cannon April 10, 2016

April 10, 2016

After a rather extensive discussion on python-ideas about pathlib.PurePath not inheriting from str, another point that came up was that the use of pathlib has been rather light. Unfortunately even the stdlib doesn't really use pathlib because it's currently marked as provisional (or at least that's why I haven't tried to use it where possible in importlib). Do we have a plan of what is required to remove the provisional label from pathlib?

36 138

pathlib (was: Defining a path protocol)
by Jim J. Jewett April 10, 2016

April 10, 2016

(1) I think the "built-in" should instead be a module-level function in the pathlib. If you aren't already expecting pathlib paths, then you're just expecting strings to work anyhow, and a builtin isn't likely to be helpful. (2) I prefer that the function be explicit about the fact that it is downcasting the representation to a string. e.g., pathlib.path_as_string(my_path) But if the final result is ospath or fspath or ... I won't fight too hard, particularly since the output may be a … [View More]

3 2

Incomplete Internationalization in Argparse Module
by Grady Martin April 9, 2016

April 9, 2016

Hello, all. I was wondering if the following string was left untouched by gettext for a purpose (from line 720 of argparse.py, in class ArgumentError): 'argument %(argument_name)s: %(message)s' There may be other untranslatable strings in the argparse module, but I have yet to encounter them in the wild. Thank you.

3 5

Re: [Python-Dev] Defining a path protocol (was: When should pathlib stop being provisional?)
by Koos Zevenhoven April 9, 2016

April 9, 2016

Nick Coghlan wrote: > On 7 April 2016 at 03:26, Brett Cannon <br...(a)python.org> wrote: >> >> Name: __path__, __fspath__, or something else? > > __fspath__ > I think I might like this dunder name because it does not clutter the list of regular methods and attributes, and is perhaps more pythonic. >> Method or attribute? (changes what kind of one-liner you might use in >> libraries, but I think historically all protocols have been methods and the >… [View More]> serialized string representation might be costly to build) > > Method, as long as there's a helper function somewhere As a further minor benefit of it being a method, it may be easier to distinguish it from from `__path__`, which is an iterable attribute. >> Built-in? (name is dependent on #1 if we add one) > > os.fspath (alongside os.fsencode and os.fsdecode) > > (Putting this in a module low in the dependency stack makes it easy > for other modules to access without pulling in all of pathlib's > dependencies) Strong +1 on putting it in os. This should also be implemented in DirEntry, instances of which are "yielded" by os.scandir. Also, you have a strong case regarding naming with the 'fs' prefix. It is also easier to read fspath as f-s-path than it is to read ospath as o-s-path, because ospath could also be pronounced as a single (meaningless?) word. I'm still thinking a little bit about 'pathname', which to me sounds more like a string than fspath does [1]. It would be nice to have the string/path distinction especially when pathlib adoption grows larger. But who knows, maybe somewhere in the far future, no-one will care much about fspath, fsencode, fsdecode or os.path. >> Add the method/attribute to str? (I assume so, much like __index__() is on >> int, but I have not seen it explicitly stated so I would rather clarify it) > > Makes sense If added to str, it should also be added to bytes. But will that then return str or bytes? See also the next point. > Expand the C API to have something like PyObject_Path()? > > PyUnicode_FromFSPath, perhaps? The return type is well-defined here, > so it can be done as an alternate constructor, and the C API > counterparts of os.fsdecode and os.fsencode are PyUnicode functions > (specifically PyUnicode_DecodeFSDefault and PyUnicode_EncodeFSDefault) What about DirEntry, which may have a bytes representation? I would expect the function return type of os.fspath to be Union[str, bytes], unless bytes pathnames are decoded with surrogate escapes. [1] https://mail.python.org/pipermail/python-ideas/2016-April/039595.html PS. I have been reading this list occasionally on the google groups mirror, and I now subscribed to it just to send this. (BTW, I probably broke the thread, as I did not have Nick's email in my inbox to reply to. Sorry about that.) I'll have to mention that I was surprised, to say the least, to find that the pathlib discussion had moved here from python-ideas, where I had mentioned I was working on a proposal. Then, I also found that the solution discussed here was seemingly an improved version of what I had proposed on python-ideas somewhat earlier [1], but did not get any reactions to. While I can only make guesses about what happened, these kinds of things easily make you go from "Hey, maybe I'll be able to do something to improve Python!" to "These people don't seem to want me here or appreciate my efforts.". Not to accuse anyone in particular; just to let people know. Anyway, I somehow got sucked into thinking deeply about pathlib etc. (which I do use). Not that I really have much at stake here, except spending ridiculous amounts of time thinking about paths, mainly during my Easter holidays and after that. I really had a hard time explaining to friends and family what the heck I was doing ;). [View Less]

6 9

Question about the current implementation of str
by Larry Hastings April 9, 2016

April 9, 2016

I have a straightforward question about the str object, specifically the PyUnicodeObject. I've tried reading the source to answer the question myself but it's nearly impenetrable. So I was hoping someone here who understands the current implementation could answer it for me. Although the str object is immutable from Python's perspective, the C object itself is mutable. For example, for dynamically-created strings the hash field may be lazy-computed and cached inside the object. I was … [View More]

4 4

Summary of Python tracker Issues
by Python tracker April 8, 2016

April 8, 2016

ACTIVITY SUMMARY (2016-04-01 - 2016-04-08) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5477 ( +6) closed 32993 (+22) total 38470 (+28) Open issues with patches: 2381 Issues opened (23) ================== #26686: email.parser stops parsing headers too soon. http://bugs.python.org/issue26686 opened by msapiro #26687: Use Py_RETURN_NONE in … [View More]

1 0

Other pathlib improvements? was: When should pathlib stop being provisional?
by Eric Snow April 8, 2016

April 8, 2016

On Apr 6, 2016 11:11 PM, "Raymond Hettinger" <raymond.hettinger(a)gmail.com> wrote: > Having worked through the API when it is first released, I find it to be highly forgettable (i.e. I have to re-read the docs each time I've revisited it). Agreed, though it's arguably better than argparse, logging, unittest, or several other stdlib modules. To some extent the challenge with those is the complexity of the problem space. Furthermore, the key for any sufficiently complex module is … [View More]

7 7

summary: a Path protocol
by Ethan Furman April 8, 2016

April 8, 2016

The discussion has ranged all over, so let me try to sum up: Name: __ospath__ Method or attribute? Method (implementations are of course free to pre-build and/or cache the value) Built-in? no, rather a function in pathlib - ospath() Add the method/attribute to str? Not necessary -- but if somebody else wants to do that part I am not opposed Expand the C API to have something like PyObject_Path()? Yes - and if I understood correctly this function will do the … [View More]

3 2

Re: [Python-Dev] When should pathlib stop being provisional?
by Eric Snow April 7, 2016

April 7, 2016

On Apr 6, 2016 14:00, "Barry Warsaw" <barry(a)python.org> wrote: > Aside from the name of the attribute (though I'm partial to __path__), Ahem, pkg.__path__. -eric

1 0

Re: [Python-Dev] When should pathlib stop being provisional?
by Christophe Bal April 7, 2016

April 7, 2016

As a simple user, pathlib simplifies playing with paths. A lot of things are easy to do. For example, Pathlib / "subfile" is so useful. I also have a subclass of pathlib.Path on github that makes easy seeking for files and directories. So keep alive pathlib ! Le 6 avr. 2016 13:06, "Paul Moore" <p.f.moore(a)gmail.com> a écrit : On 6 April 2016 at 00:45, Guido van Rossum <guido(a)python.org> wrote: > This does sound like it's the crucial issue, and it is worth writing > up … [View More]

1 0

Re: [Python-Dev] When should pathlib stop being provisional?
by Wes Turner April 6, 2016

April 6, 2016

On Apr 6, 2016 1:26 AM, "Chris Angelico" <rosuav(a)gmail.com> wrote: > > On Wed, Apr 6, 2016 at 3:37 PM, Stephen J. Turnbull <stephen(a)xemacs.org> wrote: > > Chris Angelico writes: > > > > > Outside of deliberate tests, we don't create files on our disks > > > whose names are strings of random bytes; > > > > Wishful thinking. First, names made of control characters have often > > been deliberately used by miscreants to … [View More]conceal their warez. Second, > > in some systems it's all too easy to create paths with components in > > different locales (the place I've seen it most frequently is in NFS > > mounts). I think that's much less true today, but perhaps that's only > > because my employer figured out that it was much less pain if system > > paths were pure ASCII so that it mostly didn't matter what encoding > > users chose for their subtrees. > > Control characters are still characters, though. You can take a > bytestring consisting of byte values less than 32, decode it as UTF-8, > and have a series of codepoints to work with. > > If your employer has "solved" the problem by restricting system paths > to ASCII, that's a fine solution for a single system with a single > ASCII-compatible encoding; a better solution is to mandate UTF-8 as > the file system encoding, as that's what most people are expecting > anyway. > > > It remains important to be able to handle nearly arbitrary bytestrings > > in file names as far as I can see. Please note that 100 million > > Japanese and 1 billion Chinese by and large still prefer their > > homegrown encodings (plural!!) to Unicode, while many systems are now > > defaulting filenames to UTF-8. There's plenty of room remaining for > > copying bytestrings to arguments of open and friends. > > Why exactly do they prefer these other encodings? Are they > representing characters that Unicode doesn't contain? If so, we have a > fundamental problem (no Python program is going to be able to cope > with these, without a third party library or some stupid mess of local > code); if not, you can always represent it as Unicode and encode it as > UTF-8 when it reaches the file system. Re-encoding is something that's > easy when you treat something as text, and impossible when you treat > it as bytes. > > So far, you're still actually agreeing with me: paths are *text*, but > sometimes we don't know the encoding (and that's a problem to be > solved). re: bytestring, unicode, encodings after e.g. os.path.split / Path.split: from "[Python-ideas] Type hints for text/binary data in Python 2+3 code" https://mail.python.org/pipermail/python-ideas/2016-March/038869.html >> would/will it be possible to use Typing.Text as a base class for even-more abstract string types https://mail.python.org/pipermail/python-ideas/2016-March/039016.html >> * Text.encoding >> * Text.lang (urn:ietf:rfc:3066) ... forgot to CC: >> * https://tools.ietf.org/html/rfc5646 "Tags for Identifying Languages" urn:ietf:rfc:5646 is this (Path) a narrower case of string types (#strypes), because after transformations we want to preserve string metadata like e.g encoding? I'd vote for * adding DirEntry.__path__ as a proxy to DirEntry.path * standardizing on __path__ (over .path) * because this operation *is* fundamentally similar to e.g. __str__ * operator.path pathify, pathifize > > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev(a)python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com [View Less]

1 0

GSoC: looking for a student to help on FAT Python
by Victor Stinner April 5, 2016

April 5, 2016

Hi, I am now looking for a Google Summer of Code (GSoC) student to help me of my FAT Python project, a new static optimizer for CPython 3.6 using specialization with guards. The FAT Python project is already fully functional, the code is written and tested. I need help to implement new efficient optimizations to "finish" the project and prove that my design allows to really run applications faster. FAT Python project: https://faster-cpython.readthedocs.org/fat_python.html fatoptimizer … [View More]

3 2

Py_SETREF vs. Py_XSETREF
by Serhiy Storchaka April 4, 2016

April 4, 2016

Originally I proposed a pair of macros for safe reference replacing to reflects the duality of Py_DECREF/Py_XDECREF. [1], [2] The one should use Py_DECREF and the other should use Py_XDECREF. But then I got a number of voices for the single name [3], and no one voice (except mine) for the pair of names. Thus in final patches the single name Py_SETREF that uses Py_XDECREF is used. Due to adding some overhead in comparison with using Py_DECREF, this macros is not used in critical … [View More]

6 5