Mailman 3 April 2016 - Python-Dev

Tag-based buildmaster (was: Most 3.x buildbots are green again ... )
by Zachary Ware April 13, 2016

April 13, 2016

(Cross-posting to python-buildbots, discussion is probably best continued there) On Wed, Apr 13, 2016 at 3:37 PM, Brett Cannon <brett(a)python.org> wrote: > On Wed, 13 Apr 2016 at 13:17 Zachary Ware <zachary.ware+pydev(a)gmail.com> > wrote: >> After receiving a suggestion from koobs several months ago, I've been >> intermittently thinking about completely redoing our buildmaster setup >> such that instead of a single builder per version on each slave, we >> instead set up a series of builders with particular 'tags', and each >> builder attaches to each slave that satisfies the tags (running each >> build only on the first slave available). This would allow us to test >> some of the rarer options (such as --without-threads) significantly >> more often than 'never', and generally get a lot more >> customization/flexibility of builds. I haven't had a chance to sit >> down and think out all the edge cases of this idea, but what do people >> generally think of it? I think the GitHub switchover will be a good >> time to do this if it's generally seen as a decent idea, since there >> will need to be some work on the buildmaster to do the switch anyway. > > So we have slaves connect to multiple builders who have requirements of what > they are testing? So the --without-threads master would have all slaves able > to compile --without-threads connect to it and then do that build? And those > same slaves may also connect to the gcc and clang masters to do those builds > as well? So would that mean slaves could potentially do a bunch of builds > per change? That sounds nice to me as long as the slave maintainers are also > up to utilizing this by double/triple/quadrupling their builds. Basically, yes. I'm unsure as to whether the build would be done on all matching slaves on each change, or rotate between them (or use the next available) on each change; that would likely come down to which scheme we collectively want. I also have vague ideas about having 'daily' or even 'weekly' tags for builds that are deemed to not need a build for every changeset, which could alleviate some of the multiplying. -- Zach

1 0

ping on issue 18378: locale.getdefaultlocale() fails on recent Mac OS X
by Chris Barker April 12, 2016

April 12, 2016

Hi folks, There have been multiple reports of folks having failures on startup of matplotlib, which appears to be due to the most recent OS-X version setting the locale weirdly. This was identified last summer in this issue: http://bugs.python.org/issue18378 It looks like the issue was figured out, and even a patch contributed, but it stalled out before being applied. I have no idea if the patch is any good, but it would be great to get this fixed! -Thanks, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

1 0

Maybe, just maybe, pathlib doesn't belong.
by Alexander Walters April 12, 2016

April 12, 2016

In reviewing the ongoing arguments about how to make pathlib better, there have been circular arguments about if it is even broken, if it should support bytes, if there should be a path protocol that all functions that touch the filesystem should use, if that protocol should support bytes, how that protocol should be open or closed to allow third party modules to act as paths, etc., etc. If there is headway being made, I do not see it. I don't think we can come to an agreement that will make anyone happy, or have any effect on the adoption of the pathlib module in the standard library. Maybe, just maybe, since there is an ecosystem of third party modules already doing this job (and arguably doing it much better than pathlib, and for more supported versions of python than any future version of pathlib will), it should be dropped from the standard library and left on pypi as a third party module.

13 31

Challenge: Please break this! (a.k.a restricted mode revisited)
by Jon Ribbens April 12, 2016

April 12, 2016

I've made another attempt at Python sandboxing, which does something which I've not seen tried before - using the 'ast' module to do static analysis of the untrusted code before it's executed, to prevent most of the sneaky tricks that have been used to break out of past attempts at sandboxes. In short, I'm turning Python's usual "gentleman's agreement" that you should not access names and attributes that are indicated as private by starting with an underscore into a rigidly enforced rule: try and access anything starting with an underscore and your code will not be run. Anyway the code is at https://github.com/jribbens/unsafe It requires Python 3.4 or later (it could probably be made to work on Python 2.7 as well, but it would need some changes). I would be very interested to see if anyone can manage to break it. Bugs which are trivially fixable are of course welcomed, but the real question is: is this approach basically sound, or is it fundamentally unworkable?

20 70

Re: [Python-Dev] Pathlib enhancements - acceptable inputs and outputs for __fspath__ and os.fspath()
by INADA Naoki April 12, 2016

April 12, 2016

Sorry, I've forgot to use "Reply All". On Tue, Apr 12, 2016 at 9:49 AM, INADA Naoki <songofacandy(a)gmail.com> wrote: > IHMO it's safer to get an encoding error rather than no error when you >> concatenate two byte strings encoded to two different encodings (mojibake). >> >> print(os.fspath(obj)) will more likely do what you expect if os.fspath() >> always return str. I mean that it will encode your filename to the encoding >> of the terminal which can be different than the filesystem encoding. >> >> If fspath() can return bytes, you should write >> print(os.fsdecode(os.fspath(obj))). >> >> > Why not print(obj)? > str() is normal high-level API, and __fspath__ and os.fspath() should be > low level API. > Normal users shouldn't use __fspath__ and os.fspath(). Only library > developers should use it. > > -- > INADA Naoki <songofacandy(a)gmail.com> > -- INADA Naoki <songofacandy(a)gmail.com>

2 1

thoughts on backporting __wrapped__ to 2.7?
by Robert Collins April 12, 2016

April 12, 2016

I'm working on teaching funcsigs - the backport of inspect.signature - better handling for wrapped functions, and the key enabler to do that is capturing the wrapped function in __wrapped__. I'm wondering what folks thoughts are on backporting that to 2.7 - seems cleaner than monkeypatching functools.wraps, which would tend to be subject to import ordering races and general ick. I'll likely prep such a monkeypatch for folk that are stuck on older versions of 2.7 anyhow... so its not a huge win... -Rob -- Robert Collins <rbtcollins(a)hpe.com> Distinguished Technologist HP Converged Cloud

4 5

pathlib+os/shutil feedback
by Sven R. Kunze April 11, 2016

April 11, 2016

I talked to my colleague. He didn't remember the concrete use-case, though, he instantly mentioned three possible things (no order of preference): 1) pathlib + mtime 2) os.walk and pathlib 3) creation/removal of paths He wasn't too sure but I checked with the docs and his memories seemed to be correct: ----- 1) https://docs.python.org/3/library/pathlib.html#pathlib.Path.stat High-level path objects should return high-level [insert type here] objects. Put differently, an API for retrieving time-stats as real date/time objects would be nice. I think that can be expanded to other pathlib methods as well, to make them less "os-wrapper"-like and provide added value. ----- 2) I remember a discussion on python-ideas about using "glob" or "rglob". However, when searching the docs for "walk" like in "os.walk" or for "iter", I don't find "glob"/"rglob". I can imagine ourselves (pathlib newbies back then), we didn't discover them. It would be great if the docs could be improved like the following: """ Path.rglob(pattern) Walk down a given path; a wrapper for "os.scandir"/"os.listdir". This is like calling glob() with “**” added in front of the given pattern: """ I think it would make "glob" and "rglob" more discoverable to new users. NOTE: """ Using the “**” pattern in large directory trees may consume an inordinate amount of time.""" sounds not really encouraging. This is especially true for "rglob" as it is defined as "like calling glob() with “**”". That leads to wondering whether "rglob" performs slow globbing instead of a "os.scandir"/"os.listdir". https://docs.python.org/3/library/pathlib.html#basic-use even promotes "glob" with "**" in the beginning which seems rather discouraging to use "rglob" as a fast alternative to "os.walk/scandir/listdir". Renaming "rglob"/adding a "scan" method would definitely help here. ----- 3) Again searching the docs for "create", "delete" (nothing found) and "remove", I found "Path.touch", "Path.rmdir" and "Path.unlink". It would be great if we had an easy way to remove a complete subtree as with "shutil.rmtree". We mostly don't care if a directory is empty. We need the system to be in a state of "this path does not exist anymore". Moreover, touching a file is good enough to "create" it if you don't care about changing its mtime. It you care about its mtime, it's a problem to "touch". ------ That's it for our issues with pathlib from the past. Additionally, I got two further observations: A) pathlib tries to mimic/publish some low-level APIs to its users. "unlink" is not something people would expect to use when they want to "delete" or to "remove" a file or a directory. I know where the term stems from but it's the wrong layer of abstraction IMHO. Same for "touch" or "chmod". B) "rename" vs "replace". The difference is not really clear from the docs. You need to read "Path.replace" in order to understand "Path.rename" completely. (one raises an exception, the other don't if I read it correctly). If there's some agreement to change things with respect to those 5 points, I am willing to put some time into it. Best, Sven

3 3

Re: [Python-Dev] [Python-checkins] cpython (2.7): Issue #25910: Fixed more links in the docs.
by Tim Golden April 11, 2016

April 11, 2016

On 11/04/2016 15:38, serhiy.storchaka wrote: > - <http://www.openssl.org/docs/apps/ciphers.html#CIPHER_LIST_FORMAT>`__. > + <http://www.openssl.org/docs/apps/ciphers.html#CIPHER-LIST-FORMAT>`__. Is there any intended irony in our link to openssl not being via https? :) TJG

2 1

Pathlib enhancments - method name only
by Ethan Furman April 11, 2016

April 11, 2016

On 04/08/2016 10:46 AM, Koos Zevenhoven wrote: > On Fri, Apr 8, 2016 at 7:42 PM, Chris Barker wrote: >> On Fri, Apr 8, 2016 at 9:02 AM, Koos Zevenhoven wrote: >>> I'm still thinking a little bit about 'pathname', which to me sounds >>> more like a string than fspath does. >> >> >> I like that a lot - or even "__pathstr__" or "__pathstring__" >> after all, we're making a big deal out of the fact that a path is >> *not a string*, but rather a string is a *representation* (or >> serialization) of a path. That's a decent point. So the plausible choices are, I think: - __fspath__ # File System Path -- possible confusion with Path - __fsstr__ # File System String - __fspathstr__ # File System Path String -- zero ambiguity, but # what a mouthful -- ~Ethan~

15 33

Re: [Python-Dev] Challenge: Please break this! (a.k.a restricted mode revisited)
by Anders Munch April 11, 2016

April 11, 2016

Steven D'Aprano: > although I must admit I don't understand why the entire OS is effected. A consequence of memory overcommit, I'd wager. The crasher code not only allocates vast swathes of memory, but accesses it as well, which is bad news for Linux with overcommit enabled. When the OS runs out of backing store to handle page faults, anything can happen. - Anders

1 0