Mailman 3 June 2012 - Python-ideas

for/else statements considered harmful
by Alice Bevan–McGregor 16 Jun '12

16 Jun '12

Howdy! Was teaching a new user to Python the ropes a short while ago and ran into an interesting headspace problem: the for/else syntax fails the obviousness and consistency tests. When used in an if/else block the conditional code is executed if the conditional passes, and the else block is executed if the conditional fails. Compared to for loops where the for code is repeated and the else code executed if we "naturally fall off the loop". (The new user's reaction was "why the hoek would I ever use for/else?") I forked Python 3.3 to experiment with an alternate implementation that follows the logic of pass/fail implied by if/else: (and to refactor the stdlib, but that's a different issue ;) for x in range(20): if x > 10: break else: pass # we had no values to iterate finally: pass # we naturally fell off the loop It abuses finally (to avoid tying up a potentially common word as a reserved word like "done") but makes possible an important distinction without having to perform potentially expensive length calculations (which may not even be possible!) on the value being iterated: that is, handling the case where there were no values in the collection or returned by the generator. Templating engines generally implement this type of structure. Of course this type of breaking change in semantics puts this idea firmly into Python 4 land. I'll isolate the for/else/finally code from my fork and post a patch this week-end, hopefully. — Alice.

19 42

Multi-line comment blocks.
by David Gates 16 Jun '12

16 Jun '12

Multi-line strings as comments don't nest, don't play well with docstrings, and are counter-intuitive when there's special language support for single-line comments. Python should only have one obvious way to do things, and Python has two ways to comment, only one of which is obvious. My suggestion is to add language support for comment blocks, using Python's existing comment delimiter: # Single-line comment #: Multi-line comment #: Nested multi-line comments work perfectly Of course they do, they're just nested blocks def foo(): """Docstrings work perfectly. Why wouldn't they?""" pass # No need for an end-delimiter like """ or */

13 31

Weak-referencing/weak-proxying of (bound) methods
by Jan Kaliszewski 16 Jun '12

16 Jun '12

Hello, Today, I encountered a surprising bug in my code which creates some weakref.proxies to instance methods... The actual Python behaviour related to the issue can be ilustrated with the following example: >>> import weakref >>> class A: ... def method(self): print(self) ... >>> A.method <function method at 0xb732926c> >>> a = A() >>> a.method <bound method A.method of <__main__.A object at 0xb7326bec>> >>> r = weakref.ref(a.method) # creating a weak reference >>> r # ...but it appears to be dead <weakref at 0xb7327d9c; dead> >>> w = weakref.proxy(a.method) # the same with a weak proxy >>> w <weakproxy at 0xb7327d74 to NoneType at 0x829f7d0> >>> w() Traceback (most recent call last): File "<stdin>", line 1, in <module> ReferenceError: weakly-referenced object no longer exists This behaviour is perfectly correct -- but still surprising, especially for people who know little about method creation machinery, descriptors etc. I think it would be nice to make this 'trap' less painful -- for example, by doing one or both of the following: 1. Describe and explain this behaviour in the weakref module documentation. 2. Provide (in functools?) a type-and-decorator that do the same what func_descr_get() does (transforms a function into a method) *plus* caches the created method (e.g. at the instance object). A prototype implementation: class InstanceCachedMethod(object): def __init__(self, func): self.func = func (self.instance_attr_name ) = '__{0}_method_ref'.format(func.__name__) def __get__(self, instance, owner): if instance is None: return self.func try: return getattr(instance, self.instance_attr_name) except AttributeError: method = types.MethodType(self.func, instance) setattr(instance, self.instance_attr_name, method) return method A simplified version that reuses the func.__name__ (works well as long as func.__name__ is the actual instance attribute name...): class InstanceCachedMethod(object): def __init__(self, func): self.func = func def __get__(self, instance, owner): if instance is None: return self.func method = types.MethodType(self.func, instance) setattr(instance, self.func.__name__, method) return method Both versions work well with weakref.proxy()/ref() objects: >>> class B: ... @InstanceCachedMethod ... def method(self): print(self) ... >>> B.method <function method at 0xb7329d6c> >>> b = B() >>> b.method <bound method B.method of <__main__.B object at 0xb7206ccc>> >>> r = weakref.ref(b.method) >>> r <weakref at 0xb72c611c; to 'method' at 0xb736c40c (method)> >>> w = weakref.proxy(b.method) >>> w <weakproxy at 0xb7327e14 to method at 0xb736c40c> >>> w() <__main__.B object at 0xb7206ccc> What do you think about it? Cheers. *j

3 5

Re: [Python-ideas] Python-ideas Digest, Vol 67, Issue 51
by David Gates 15 Jun '12

15 Jun '12

@Robert Kern: "Multi-line string literals aren't comments. They are multi-line string literals. Unlike a comment, which does not show up in the compiled bytecode, the Python interpreter actually does something with those string literals." They have Guido's stamp of approval, and apparently the interpreter ignores them: https://twitter.com/gvanrossum/status/112670605505077248 They feel like an ugly hack to me too, though. @Robert Kern: "The main problem is that #: currently has a meaning as a line comment. This could break existing code." It could, but the only case I can see is when the comment isn't following indentation convention: #: Valid either way; next line's not indented, #: so it's not counted as part of the block. print('a') # Causes an IndentationError in existing code. #: print('b') def foo(): #: This one would break. print('c') On Fri, Jun 15, 2012 at 4:00 AM, <python-ideas-request(a)python.org> wrote: > Send Python-ideas mailing list submissions to > python-ideas(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mail.python.org/mailman/listinfo/python-ideas > or, via email, send a message with subject or body 'help' to > python-ideas-request(a)python.org > > You can reach the person managing the list at > python-ideas-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Python-ideas digest..." > > > Today's Topics: > > 1. Multi-line comment blocks. (David Gates) > 2. Re: Multi-line comment blocks. (Robert Kern) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 15 Jun 2012 02:49:18 -0600 > From: David Gates <gatesda(a)gmail.com> > To: python-ideas(a)python.org > Subject: [Python-ideas] Multi-line comment blocks. > Message-ID: > <CAG2+q8Ska2HLksy6D49wOzgBbQ6E44xLFjqQyZRnhfJTywCmKw(a)mail.gmail.com > > > Content-Type: text/plain; charset="iso-8859-1" > > Multi-line strings as comments don't nest, don't play well with docstrings, > and are counter-intuitive when there's special language support for > single-line comments. Python should only have one obvious way to do things, > and Python has two ways to comment, only one of which is obvious. My > suggestion is to add language support for comment blocks, using Python's > existing comment delimiter: > > # Single-line comment > #: > Multi-line comment > #: > Nested multi-line comments work perfectly > Of course they do, they're just nested blocks > def foo(): > """Docstrings work perfectly. Why wouldn't they?""" > pass > # No need for an end-delimiter like """ or */ >

1 0

Re: [Python-ideas] TextIOWrapper callable encoding parameter
by Rurpy 13 Jun '12

13 Jun '12

On 06/12/2012 03:48 PM, Victor Stinner wrote: >> >> 1. The most straight-forward way to handle this is to open >> >> the file twice, first in binary mode or with latin1 encoding >> >> and again in text mode after the encoding has been determined >> >> This of course has a performance cost since the data is read >> >> twice. Further, it can't be used if the data source is a >> >> from a pipe, socket or other non-rewindable source. This >> >> includes sys.stdin when it comes from a pipe. > > > > Some months ago, I proposed to automatically detect if a file contains > > a BOM and uses it to set the encoding. Various methods were proposed > > but there was no real consensus. One proposition was to use a codec > > (e.g. "bom") which uses the BOM if it is present, and so don't need to > > reread the file twice. > > > > For the pipe issue: it depends where the encoding specification is. If > > the encoding is written at the end of your "file" (stream), you have > > to store the whole stream content (few MB or maybe much more?) into > > memory. If it is in the first lines, you have to store these lines in > > a buffer. It's not easy to decide for the threshold. That's always a problem. When trying to determine a character encoding one may have to read the entire file because it could consist of all ascii characters except the very last one. (And of course there is no guarantee one can determine *the* encoding at all). Nevertheless, I think thee is a very large class of problems that can be usefully handled by looking at a limited amount of data at the start of a file (or stream). The Python coding declaration in one example (obviously picked hoping it would have some resonance here.) The buffer object used by TextIOWrapper already reads the start of the stream and buffers the first few lines, so why not take advantage of that rather than repeating the work? One of the things I am not sure about is if there are cases when the buffered read returns, say, only one line, as might happen with tty input. > > I don't like the codec approach because the codec is disconnected from > > the stream. For example, the codec doesn't know the current position > > in stream nor can read a few more bytes forward or backward. If you > > open the file in "append" mode, you are not writing at the beginning > > but at the end of the file. You may also seek at an arbitrary position > > before the first read... > > > > There are also some special cases. For example, when a text file is > > opened in write mode, the file is seekable and the file position is > > not zero, TextIOWrapper calls encoder.setstate(0) to not write the BOM > > in the middle of the file. (See also Lib/test/test_io.py for related > > tests.) A callable encoding parameter would not be terribly useful with a file opened in write or append mode, but it's behavior would be predictable: a write would result in an error because the encoding hadn't been set. A read in the middle' of the file would work the same way as at the beginning. This is probably not very useful, but is consistent. Of course one could choose to implement a callable encoding parameter such that some or all of these paths are detected at open and declared illegal then. One could prohibit the encoding call after a seek though I'm not sure there is any point to that. >> >> 2. Alternatively, with a little more expertise, one can rewrap >> >> the open binary stream in a TextIOWrapper to avoid a second >> >> OS file open. > > > > That's my favorite method because you have the full control on the > > stream. (I wrote tokenize.open). But yes, it does not work on > > non-seekable streams (e.g. pipes). > > >> >> This too seems to read the data twice and of course the >> >> seek(0) prevents this method also from being usable with >> >> pipes, sockets and other non-seekable sources. > > > > Does it really matter? You usually need to read few bytes to get the encoding. It certainly matters if input is from a pipe. Quoting from my other message: $ cat test.utf8 | python3 stdin.py reopen1 got exception: [Errno 29] Illegal seek The whole point of my suggestion was that you've already read those few bytes -- but by the time you have access to them, you've already been forced to choose an encoding. My suggestion simply defers that encoding setting until after you've had a chance to look at the bytes. >> >> 9. In other non-read paths where encoding needs to be known, >> >> raise an error if it is still None. > > > > Why not reading data until you the encoding is known instead? That's how I do it now -- open file in binary mode and read it, buffer it, determine encoding, and henceforth decode the bytes data "by hand" to text. But that's an awful lot like what TextIOWrpper does, yes? Why can't I use TextIOWrapper instead of rewriting it myself? (Yes, I know I can reopen or rewrap the binary stream but as I said, that loses the one-pass processing which breaks pipes.) >> >> I have modified a copy the _pyio module as described and >> >> the changes required seemed unsurprising and relatively >> >> few, though I am sure there are subtleties and other >> >> considerations I am missing. Hence this post seeking >> >> feedback... > > > > Can you post the modified somewhere so I can play with it? I put a diff against the Python-3.2.3 _pyio.py file at: http://pastebin.com/kZHmcBdm Much of the diff is just moving existing stuff around. The note at the bottom says: | It is in no way supposed to be a serious patch. | | It was the minimal changes I could make in order to | see if my suggestion to allow a callable encoding parameter | in TextIOWrapper was feasible, and allow some timing tests. | | I am quite sure it will not pass the Python's tests. | | It does I hope give some idea of the nature and scale of the | code changes needed to implement a callable encodign parameter.

1 0

Re: [Python-ideas] TextIOWrapper callable encoding parameter
by Rurpy 13 Jun '12

13 Jun '12

On 06/11/2012 10:24 AM, Stephen J. Turnbull wrote: > > Nick Coghlan writes: > > > > > Immediate thought: it seems like it would be easier to offer a way to > > > inject data back into a buffered IO object's internal buffer. > > > > ungetch()? What would be the TextIOWrapper api for that? > > If you're only interested in the top of the file (see below), I would > > suggest allowing only one bufferfull, and then simply rewinding the > > buffer pointer once you're done. This is one strategy used by Emacsen > > for encoding detection (for the reason pointed out by Rurpy: not all > > streams are rewindable). > > > > But is that really "easier"? It might be more general, but you still > > need to reinitialize the encoding (ie, from the trivial "binary" to > > whatever is detected), with all the hair that comes with that. I don't think there is any hair involved. In at least the _pyio version of TextIOWrapper, initializing the encoding (in the read path) consists of calling self._get_decoder(). One needs to move the few places where that is called now to nearby places that are after the raw buffer has been read but before it is decoded. There may be need for some consideration given to raising errors at the old locations in the case the callable encoding hook is not being used (to maintain complete backwards compatibility; not sure that is necessary), but I wouldn't call that hairy. Of course there may be other factors I am missing... > > > > Executive summary: > > > > ================== > > > > > > > > There is no good way to read a text file when the > > > > encoding has to be determined by reading the start > > > > of the file. A long-winded version of that follows. > > > > Scroll down the the "Proposal" section to skip it. > > > > This may be insufficiently general. Specifically, both Emacsen and vi > > allow specification of editor configuration variables at the bottom of > > the file as well as the top. I don't know whether vi allows encoding > > specs at the bottom, but Emacsen do (but only for files). > > > > I wouldn't recommend paying much attention to what Emacsen actually > > *do* when initializing a stream (it's, uh, "baroque"). Looking only at the beginning of an input stream is general enough for a large class of problems including tokenizing python source code.

1 0

TextIOWrapper callable encoding parameter
by Rurpy 13 Jun '12

13 Jun '12

Here is another issue that came up in my ongoing adventure porting to Python3... Executive summary: ================== There is no good way to read a text file when the encoding has to be determined by reading the start of the file. A long-winded version of that follows. Scroll down the the "Proposal" section to skip it. Problem: ======== When one opens a text file for reading, one must specify (explicitly or by default) an encoding which Python will use to convert the raw bytes read into Python strings. This means one must know the encoding of a file before opening it, which is usually the case, but not always. Plain text files have no meta-data giving their encoding so sometimes it may not be known and some of the file must be read and a guess made. Other data like html pages, xml files or python source code have encoding information inside them, but that too requires reading the start of the file without knowing the encoding in advance. I see three ways in general in Python 3 currently to attack this problem, but each has some severe drawbacks: 1. The most straight-forward way to handle this is to open the file twice, first in binary mode or with latin1 encoding and again in text mode after the encoding has been determined This of course has a performance cost since the data is read twice. Further, it can't be used if the data source is a from a pipe, socket or other non-rewindable source. This includes sys.stdin when it comes from a pipe. 2. Alternatively, with a little more expertise, one can rewrap the open binary stream in a TextIOWrapper to avoid a second OS file open. The standard library's tokenize.open() function does this: def open(filename): buffer = builtins.open(filename, 'rb') encoding, lines = detect_encoding(buffer.readline) buffer.seek(0) text = TextIOWrapper(buffer, encoding, line_buffering=True) text.mode = 'r' return text This too seems to read the data twice and of course the seek(0) prevents this method also from being usable with pipes, sockets and other non-seekable sources. 3. Another method is to simply leave the file open in binary mode, read bytes data, and manually decode it to text. This seems to be the only option when reading from non-rewindable sources like pipes and sockets, etc. But then ones looses the all the advantages of having a text stream even though one wants to be reading text! And if one tries to hide this, one ends up reimplementing a good part of TextIOWrapper! I believe these problems could be addressed with a fairly simple and clean modification of the io.TextIOWrapper class... Proposal ======== The following is a logical description; I don't mean to imply that the code must follow this outline exactly. It is based on looking at _pyio; I hope the C code is equivalent. 1. Allow io.TextIOWrapper's encoding parameter to be a callable object in addition to a string or None. 2. In __init__(), if the encoding parameter was callable, record it as an encoding hook and leave encoding set to None. 3. The places in Io.TextIOWrapper that currently read undecoded data from the internal buffer object and decode (only methods read() and read_chunk() I think) it would be modified to do so in this way: 4. Read data from the buffer object as is done now. 5. If the encoding has been set, get a decoder if necessary and continue on as usual. 6. If the encoding is None, call the encoding callable with the data just read and the buffer object. 7. The callable will examine the data, possibly using the buffer object's peek method to look further ahead in the file. It returns the name of an encoding. 8. io.TextIOWrapper will get the encoding and record it, and setup the decoder the same way as if the encoding name had been received as a parameter, decode the read data and continue on as usual. 9. In other non-read paths where encoding needs to be known, raise an error if it is still None. Were io.TextWrapper modified this way, it would offer: * Better performance since there is no need to reread data * Read data is decoded after being examined so the stream is usable with serial datasources like pipes, sockets, etc. * User code is simplified and clearer; there is better separation of concerns. For example, the code in the "Problem" section could be written: stream = open(filename, encoding=detect_encoding): ... def detect_encoding (data, buffer): # This is still basically the same function as # in the code in the "Problem" section. ... look for Python coding declaration in first two lines of the 'data' bytes object. if not found_encoding: raise Error ("unable to determine encoding") return found_encoding I have modified a copy the _pyio module as described and the changes required seemed unsurprising and relatively few, though I am sure there are subtleties and other considerations I am missing. Hence this post seeking feedback...

6 6

stdlib crowdsourcing
by anatoly techtonik 13 Jun '12

13 Jun '12

The problem with stdlib - it is all damn subjective. There is no process to add functions and modules if you're not well-behaved and skilled in public debates and don't have really a lot of time to be a champion of your module/function. In other words - it is hard (if not impossible for 80% of Python Earth population). So, many people and projects decide to opt-out. Take a look at Twisted - a lot of useful stuff, but not in Python stdlib. So.. Provide a way for people to opt-out from core stuff, but still allow to share the changes and update code if necessary. This will require: - a local stdlib Python path convention - snippet normalization function and AST hash dumper - web site with stats - source code crawler How it works: 1. Every project maintains its own stdlib directory with functions that they feel are good to have in standard library 2. Functions are placed so that they are imported as if from standard library, but this time with stdlib prefix 3. The license for this directory is public domain to remove all legal barriers (credits are welcome, but optional) 4. Crawler (probably PyPI) scans this stdlib dir, finds functions, normalizes them, calculates hash and submits to web site 4.1 Normalization is required to find the shared function copy/pasted across different projects with different indentation level, docstrings, parameters/variable names etc. 4.2 Hash is calculated upon AST. There are at least three hashes for each entry: 4.2.1 Full hash - all docstrings and variable names are preserved, whitespace normalized 4.2.2 Stripped hash - docstrings are stripped, variable names are normalized 4.2.3 Signature hash - a mark placed in a comment above function name, either calculated from function signature or generated randomly, used for manual tracking of copy/paste e.g. pd:ac546df6b8340a92 5. Web site maintains usage and popularity staff, accepts votes on inclusion of snippets User stories: 1. "I want to find if there is a better/updated version of my function available" 1.1 I enter hash into web site search form 1.2 Site gives me a link to my snippet 1.3 I can see what people proposed to replace this function with 1.4 I can choose the function with most votes 1.5 I can flag the functions I may find irrelevant or 1.5 I can tag the functions that divert in different direction than I need to filter them 2. "I want to reuse code snippets without additional dependencies on 3rd party projects" 1.1 Just place them into my own stdlib directory 3. "I want to update code snippets when there is an update for them" 1.1 I run scanner, it extracts signature hashes, stripped hashes and looks if web-site version of signature matches normalized hash 4. "I want to see what people want to include in the next Python version" 1.1 A call for proposals is made 1.2 People place wannabe's into their stdlib dirs 1.3 Crawl generates new functions on a web site 1.4 Functions are categorized 1.5 Optionally included / declined with a short one-liner reason - why 1.6 Optionally provided with more detailed info why --- feature creep cut --- 5. "I want to see what functions are popular in other languages" 1.1 A separate crawler for Ruby, PHP etc. stdlib converts their AST into compatible format where possible 1.2 Submit to site stats 6. "I want to download the function in Ruby format" 1.1 AST converter tries to do the job automatically where possible 1.2 If it fails - you are encouraged to fix the converter rules or write the replacement for this signature manually Just an idea. -- anatoly t.

9 10

Replacing the standard IO streams (was Re: changing sys.stdout encoding)
by Nick Coghlan 12 Jun '12

12 Jun '12

So, after much digging, it appears the *right* way to replace a standard stream in Python 3 after application start is to do the following: sys.stdin = open(sys.stdin.fileno(), 'r', <new settings>) sys.stdout = open(sys.stdout.fileno(), 'w', <new settings>) sys.stderr = open(sys.stderr.fileno(), 'w', <new settings>) Ditto for the other standard streams. It seems it already *is* as simple as with any other file, we just collectively forgot about: 1. The fact open() accepts file descriptors directly in Python 3 2. The fact that text streams still report the underlying file descriptor correctly *That* is something we can happily advertise in the standard library docs. If you could check to make sure it works properly for your use case and then file a docs bug at bugs.python.org to get it added to the std streams documentation, that would be very helpful. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

9 24

Re: [Python-ideas] changing sys.stdout encoding
by Rurpy 12 Jun '12

12 Jun '12

On 06/07/2012 03:45 PM, Nick Coghlan wrote: > The interpreter uses the standard streams internally, and > they're one of the first things created during interpreter > startup. User provided code doesn't start running until well > after they're initialised. In other words, the stream objects referenced by sys.std* are opened before the user code runs? But if there are no operations on those streams until my user code runs, they are still in the same state they were after they were initialized, yes? So if one wanted to provide an "only before first use" set_encoding() function, why couldn't that function reexecute the codecs part of the initialization code a second time? Of course there would need to be some sort of flag that it could use to verify the stream was still in its initial state. > If user level code doesn't want those streams, it needs to > replace them with something else. Yes, this is what the code I googled up does: import codecs sys.stdout = codecs.getwriter(opts.encoding)(sys.stdout.buffer) But that code is not obvious to someone who has been able to do all his encoded IO (with the exception of sys.stdout) using just the encoding parameter of open(). Hence my question if some- thing like a set_encoding() method/function that would work on sys.stdout is feasible. I don't see an answer to that in your statement above.

4 5