Mailman 3 July 2014 - Python-ideas

Expose __abstractmethods__/__isabstractmethod__ in abc
by Andrew Barnert 26 Jul '14

26 Jul '14

An ABC built with abc.ABCMeta has a member __abstractmethods__, which is an iterable of all of the abstract methods defined in that ABC that need to be overridden. A method decorated with @abstractmethod gets a member __isabstractmethod__=True, which is how the ABC (or, rather, the interpreter) checks whether each of its abstract methods have been overridden. However, they're part of a private protocol used by the CPython implementation of the abc module, which means any third-party code that uses them isn't portable or future-proof. Which is a shame, because there are all kinds of things you can build easily on top of abc with them, but would have to duplicate most of the module (and the special interpreter support for it) without them. The simplest change is just to document these two members as part of the module interface. Alternatively, there could be functions abc.isabstractmethod(method) and abc.abstractmethods(cls), which would allow for implementations that didn't use the same protocol internally but supported the same interface. Examples where this could be useful: * Write a runtime check-and-register function. * Explicitly test that a set of classes have implemented their ABC(s) without having to know how to correctly instantiate them. * Write a generic @autoabc decorator or similar for creating ABCs that are automatically virtual base classes of any type with the right methods, instead of doing it manually in each class (as collections.abc does today), like Go interfaces, C++ auto concepts, traditional ObjC checked informal protocols, etc.). * Build a signature-checking (rather than just name-checking) ABC (like https://github.com/apieum/ducktype). * Build a simplified version of PyProtocols-like adapters on top of abc instead of PyProtocols. Some of these might belong in the stdlib (in fact, it looks like http://bugs.python.org/issue9731 basically covers the first two), in which case they don't need to be implementable from outside… but that's certainly not true for all of them. (Without a precise algorithm for "compatible signature" or a standardized notion of adaptation, stdlib inclusion isn't even sensible for the last two, much less a good idea.) So, outside libraries should be able to implement them.

2 2

os.path.argparse - optional startdir argument
by Wolfgang Maier 25 Jul '14

25 Jul '14

Dear all, currently, os.path.abspath(somepath) is, essentially, equivalent to os.path.normpath(os.path.join(os.getcwd(),path)). However, I'd find it useful, occasionally, to be able to specify a starting directory other than the current working directory. One such situation is when reading a config file of an application: if you encounter a relative link in such a file, you'll typically want to transform it into an absolute path using that application's working directory as opposed to your own one. My suggestion would be to add an optional startdir argument to abspath, which, when provided would be used instead of os.getcwd(). If startdir itself is not an absolute path either, it would be turned into one through recursion. Currently, you have to write: os.path.normpath(os.path.join(startdir, path)) or even os.path.normpath(os.path.join(os.path.abspath(startdir), path)) instead of the proposed: os.path.abspath(path, startdir) Before posting I checked the bug tracker and found that this idea has been brought up years ago (http://bugs.python.org/issue9882), but not pursued further. The patch suggested there is a bit of an oversimplification, but I have my own one, which I could provide if someone's interested. For issue9882 it was suggested to bring it up on python-ideas, but to the best of my knowledge that was never done, so I'm doing it now. Thoughts ? Wolfgang

8 18

os.path.cansymlink(path)
by anatoly techtonik 25 Jul '14

25 Jul '14

This is a live code from current virtualenv.py: if hasattr(os, 'symlink'): logger.info('Symlinking Python bootstrap modules') This code is wrong, because OS support for symlinks doesn't guarantee that mounted filesystem can do this, resulting in OSError at runtime. So, the proper check would be to check if specific path supports symlinking. The idea is: os.path.cansymlink(path) - Return True if filesystem of specified path can be symlinked. Yes/No/Opinions? -- anatoly t.

7 9

Iterating non-newline-separated files should be easier
by Andrew Barnert 23 Jul '14

23 Jul '14

tl;dr: readline and friends should take an optional sep parameter (which also means adding an iterlines method). Recently, I was trying to add -0 support to a command-line tool, which means that it reads filenames out of stdin and/or a text file with \0 separators instead of \n. This means that my code that looked like this: with open(path, encoding=sys.getfilesystemencoding()) as f: for filename in f: do_stuff(filename) … turned into this (from memory, not the exact code): def resplit(chunks, sep): buf = b'' for chunk in chunks: parts = (buf+chunk).split(sep) yield from parts[:-1] buf = parts[-1] if buf: yield buf with open(path, 'rb') as f: chunks = iter(lambda: f.read(4096), b'') for line in resplit(chunks, b'\0'): filename = line.decode(sys.getfilesystemencoding()) do_stuff(filename) Besides being a lot more code (and involving things that a novice might have problems reading like that two-argument iter), this also means that the file pointer is way ahead of the line that's just been iterated, I'm inefficiently buffering everything twice, etc. The problem is that readline is hardcoded to look for b'\n' for binary files, smart-universal-newline-thingy for text files, there's no way to reuse its machinery if you want to look for something different, and there's no way to access the internals that it uses if you want to reimplement it. While it might be possible to fix the latter problems in some generic and flexible way, that doesn't seem all that useful; really, other than changing the way readline splits, I don't think anyone wants to hook anything else about file objects. (On the other hand, people might want to hook it in more complex ways—e.g., pass a separator function instead of a separator string? I'm probably reaching there…) If I'm right, all that's needed is an extra sep=None keyword-only parameter to readline and friends (where None means the existing newline behavior), along with an iterlines method that's identical to __iter__ except that it has room for that new parameter. One minor side problem: Sometimes you don't actually have a file, but some kind of file-like object. I realize that as 3.1 or so, this is supposed to mean it actually is an io.BufferedIOBase or etc., but there are still plenty of third-party modules that just demand and/or provide "something with read(size)" or the like. In fact, that's the case with the problem I ran into above; another feature uses a third-party module to provide file-like objects for members of all kinds of uncommon archive types, and unlike zipfile, that module wasn't changed to provide io subclasses when it was ported to 3.x. So, it might be worth having adapters that make it easier (or just possible…) to wrap such a thing in the actual io interfaces. (The existing wrappers aren't adapters—BufferedReader demands readinto(buf), not read(size); TextIOWrapper can only wrap a BufferedIOBase.) But that's really a separate issue (and the answer to that one may just be to hold firm with the "file-like object means IOBase" and eventually every library you care about will work that way, even if you occasionally have to fix it yourself).

17 58

Struct format with multiple endianness markers
by Greg Ewing 22 Jul '14

22 Jul '14

I'd like to propose a small enhancement to the struct module: Allow the endianness characters to occur more than once in the format string, rather than just as the first character. My use case is reading ESRI shapefile headers, which mix big and little endian data. This means I can't use a single struct.unpack call to read what is logically a single structure, but have to split it up and use multiple calls. If I could switch endianness part way through the format, I could unpack the whole structure with a single call. -- Greg

2 1

Changing `Sequence.__contains__`
by Ram Rachum 21 Jul '14

21 Jul '14

Why does the default `Sequence.__contains__` iterate through the items rather than use `.index`, which may sometimes be more efficient? I suggest an implementation like this: def __contains__(self, i): try: self.index(i) except ValueError: return False else: return True What do you think?

5 9

lazy tuple unpacking
by Paul Tagliamonte 08 Jul '14

08 Jul '14

Given: >>> def g_range(n): ... for y in range(n): ... yield y ... I notice that: >>> a, b, c, *others = g_range(100) Works great. Super useful stuff there. Looks good. I also notice that this causes *others to consume the generator in a greedy way. >>> type(others) <class 'list'> And this makes me sad. >>> a, b, c, *others = g_range(10000000000) # will also make your machine very sad. Eventually resulting # (ok, unless you've got a really fancy bit of kit) in: Killed Really, the behavior (I think) should be more similar to: >>> _x = g_range(1000000000000) >>> a = next(_x) >>> b = next(_x) >>> c = next(_x) >>> others = _x >>> Of course, this leads to all sorts of fun errors, like the fact you couldn't iterate over it twice. This might not be expected. However, it might be nice to have this behavior when you're unpacking a generator. Thoughts? Paul

5 8

PEP pre-draft: Support for indexing with keyword arguments
by Stefano Borini 05 Jul '14

05 Jul '14

Dear all, after the first mailing list feedback, and further private discussion with Joseph Martinot-Lagarde, I drafted a first iteration of a PEP for keyword arguments in indexing. The document is available here. https://github.com/stefanoborini/pep-keyword/blob/master/PEP-XXX.txt The document is not in final form when it comes to specifications. In fact, it requires additional discussion about the best strategy to achieve the desired result. Particular attention has been devoted to present alternative implementation strategies, their pros and cons. I will examine all feedback tomorrow morning European time (in approx 10 hrs), and apply any pull requests or comments you may have. When the specification is finalized, or this community suggests that the PEP is in a form suitable for official submission despite potential open issues, I will submit it to the editor panel for further discussion, and deploy an actual implementation according to the agreed specification for a working test run. I apologize for potential mistakes in the PEP drafting and submission process, as this is my first PEP. Kind Regards, Stefano Borini

19 39

Re: [Python-ideas] PEP pre-draft: Support for indexing with keyword arguments
by Stefano Borini 04 Jul '14

04 Jul '14

On Fri, Jul 04, 2014 at 08:20:18PM +0200, Oleg Broytman wrote: > On Fri, Jul 04, 2014 at 08:10:51PM +0200, Stefano Borini <stefano.borini(a)ferrara.linux.it> wrote: > > C1. a[Z=3] -> idx = {"Z": 3} > > C2. a[Z=3, R=4] -> idx = {"Z"=3, "R"=4} > > Huh? Shouldn't it be > C2. a[Z=3, R=4] -> idx = {"Z": 3, "R": 4} yes. typo. already fixed in the PEP > > Cons: > > - degeneracy of a[{"Z": 3, "R": 4}] with a[Z=3, R=4], but the same degeneracy exists > > for a[(2,3)] and a[2,3]. > > There is no degeneration in the second case. Tuples are created by > commas, not parentheses (except for an empty tuple), hence (2,3) and 2,3 > are simply the same thing. We discussed this point above in the thread, and you are of course right in saying so, yet it stresses the fact that no matter what you pass inside those square brackets, they always end up funneled inside a single object, which happens to be a tuple that you just created > While Z=3, R=4 is far from being the same as > {"Z": 3, "R": 4}. but dict(Z=3, R=4) is the same as {"Z": 3, "R": 4}. this is exactly like tuple((2,3)) is the same as (2,3) See the similarity? the square brackets "call a constructor" on its content. This constructor is tuple if entries are not key=values (except for the single index case, of course), and dict if entries are key=values.

5 9

Re: [Python-ideas] Empty set, Empty dict
by Wichert Akkerman 01 Jul '14

01 Jul '14

Victor Stinner wrote: 2014-06-10 8:15 GMT+02:00 Neil Girdhar <mistersheik at gmail.com>: > > I've seen this proposed before, and I personally would love this, but my > > guess is that it breaks too much code for too little gain. > > > > On Wednesday, May 21, 2014 12:33:30 PM UTC-4, Frédéric Legembre wrote: > >> > >> > >> Now | Future | > >> ---------------------------------------------------- > >> () | () | empty tuple ( 1, 2, 3 ) > >> [] | [] | empty list [ 1, 2, 3 ] > >> set() | {} | empty set { 1, 2, 3 } > >> {} | {:} | empty dict { 1:a, 2:b, 3:c } > > > Your guess is right. It will break all Python 2 and Python 3 in the world. > > Technically, set((1, 2)) is different than {1, 2}: the first creates a > tuple and loads the global name "set" (which can be replaced at > runtime!), whereas the later uses bytecode and only store values > (numbers 1 and 2). > > It would be nice to have a syntax for empty set, but {} is a no-no. Perhaps {,} would be a possible spelling. For consistency you might want to allow (,) to create an empty tuple as well; personally I would find that more intuitive that (()). Wichert.

18 47