Mailman 3 June 2012 - Python-ideas

Re: [Python-ideas] changing sys.stdout encoding
by Rurpy 12 Jun '12

12 Jun '12

On 06/07/2012 03:00 PM, Mike Meyer wrote: > On Thu, Jun 7, 2012 at 4:48 PM, Rurpy <rurpy-/E1597aS9LQAvxtiuMwx3w(a)public.gmane.org> wrote: >> I suspect the vast majority of >> programmers are interested in a language that allows >> them to *effectively* get done what they need to, whether >> they are working of the latest agile TTD REST server, or >> modifying some legacy text files. > > Others have raised the question this begs to have answered: how do > other programming languages deal with wanting to change the encoding > of the standard IO streams? Can you show us how they do things that's > so much easier than what Python does? This is how it seems to be done in Perl: binmode(STDOUT, ":encoding(sjis)"); which seems quite a bit simpler than Python. I don't know if it meets your "so much easier" criterion. A quick trial showed that it works as advertised when called before any output. The description of binmode() in "man perlfunc" sounds like encoding can be changed on-the-fly but my attempt to do so had no effect, so I don't know if I'm misinterpreting the text or wrote bad Perl code (haven't used it in ages and not interested in relearning it right now.) TCL appears to have on-the-fly encoding changes: | encoding system ?encoding? | Set the system encoding to encoding. If encoding is omitted | then the command returns the current system encoding. The system | encoding is used whenever Tcl passes strings to system calls. http://www.tcl.tk/man/tcl8.4/TclCmd/encoding.htm I'll see if I can find out about some other languages if there continues to be any interest. >> And even were I to accept your argument, Python is >> inconsistent: when I open a file explicitly there is >> only a slight penalty for opening a non-default-encoded >> file (the need the explicitly give an encoding): > > The proper encoding for the standard IO streams is generally a > property of the environment, and hence is set in the environment. "Proper encoding"? If you said, "Proper default encoding" I'd agree with you. And I'd buy your claim if no one had ever invented output redirection and if print output always went to a console with a (relatively) fixed encoding. But that is not the case. > You > have a use case where that's not the case. The argument is that your > use case isn't common enough to justify changing the standard library. > Can you provide evidence to the contrary? How exactly do you suggest one accurately quantify "commonness"? And what is the threshold for justification? It seems to me the strongest argument is the credibility one that I already made: 1) Programs that accept data input on stdin and write data on stdout have a long history and are widely used. I hope this is self evident. 2) Encodings other than utf-8 are widely used. I pointed to the commonness of non-utf8 encoding in Japanese web pages. Additionally, Google for "ftp readme の site:.jp" turns up lots of text files. Once past the first few pages of Google results (where the web pages are mostly utf8) hardly any utf8 files are to be found. 3) An effect of globalization means that many more programmers today are dealing with files that have non-native encoding that come from or go to customers, vendors, partners and colleagues in other parts of the world. The number of encodings in wide use even within a single country (again Japan: utf8, sjis, euc-jp, iso202jp) implies pretty strongly that tools for use only in that region will often need multi-encoding capabilities. I think connecting the dots above leads to a pretty high-probability conclusion. > Other languages that make > setting the encoding on the standard streams easy, or applications > outside of those built for your system that have a "--encoding" type > flag? iconv, recode and their ilk are obvious examples of applications. >> I wasn't suggesting a change to the core level (if by that >> you mean to the interpreter). I was asking if some way could >> be provided that is easier and more reliable than googling >> around for a magic incantation) to change the encoding of one >> or more of the already-open-when-my-program-starts sys.std* >> streams. I presume that would be a standard library change >> (in either the io or sys modules) and offered a .set_encoding() >> method as a placeholder for discussion. > > Why presume that this needs a change in the library? The method is > straightforward, if somewhat ugly. Is there any reason it can't just > be documented, instead of added to the library? Changing the library > would require a similar documentation change. Did you miss the paragraph right below the one you quote? The one in which I said, >> An inferior and bare minimum way to address this would be to >> at least add a note about how to change the encoding to the >> sys.std* documentation. That encourages cargo-cult programming >> and doesn't address the WTF effect but it is at least better >> than the current state of affairs.

2 1

Re: [Python-ideas] TextIOWrapper callable encoding parameter
by Rurpy 11 Jun '12

11 Jun '12

As a followup, here are some timing data that seem to confirm a modest increase in speed as a result of implementing the callable encoding parameter I proposed (although that would not be the main reason for wanting to do it.) These are just for illustration. (Among many other reasons, _pyio benchmarks are not very useful.) I read four short test files using four methods for determining the test file's encoding. The test files are a simplified model of a python coding declaration (always on first line in our case with no BOM present [*1]) followed by mixed english and japanese text. Method 0 (reopen0): Use the encoding callable I am proposing. def reopen0 (fname): def hook (data,buf): return get_encoding (data) t = io.open (fname, encoding=hook) Method 1 (reopen1): Open in binary to determine encoding, then rewrap in a TextIOWrapper with the correct encoding. def reopen1 (fname): b = io.open (fname, 'rb') line = b.readline() enc = get_encoding (line) b.seek (0) t = io.TextIOWrapper (b, enc, line_buffering=True) t.mode = 'r' Method 2 (reopen2): Open in binary to determine encoding, then reopen in text mode with correct encoding. def reopen2 (fname): b = io.open (fname, 'rb') line = b.readline() enc = get_encoding (line) t = io.open (fname, encoding=enc) Method 3 (reopen3): Open in text mode (latin1) to determine encoding, then reopen in text mode with correct encoding. def reopen3 (fname): f = io.open (fname, encoding='latin1') line = f.readline() enc = get_encoding (line) t = io.open (fname, encoding=enc) The same get_encoding() function is used in all methods [*1]. The input test data are all small files (because we want to measure encoding detection, not how fast read() runs.) Each has a python/emacs coding declaration in the first line. test.utf8 -- Tiny python program with coding declaration and single print statement in main() function that prints a short word (literal) in Japanese. Encoding is utf-8 (122 bytes). test.sjis -- Identical to test.utf8 but sjis encoding (111 bytes). test2.utf8 -- A python coding declaration followed by approximately 50 long lines with mixed English and Japanese (4274 bytes). test2.sjis -- Identical to test2.utf8 but sjis encoding (3401 bytes). Results: --------------------------------------------------------- $ python3 bm.py test.utf8 test.utf8 / reopen0: total time (10000 reps) was 1.188323 test.utf8 / reopen1: total time (10000 reps) was 1.490757 test.utf8 / reopen2: total time (10000 reps) was 1.766081 test.utf8 / reopen3: total time (10000 reps) was 2.141996 $ python3 bm.py test.sjis test.sjis / reopen0: total time (10000 reps) was 1.175914 test.sjis / reopen1: total time (10000 reps) was 1.471780 test.sjis / reopen2: total time (10000 reps) was 1.764444 test.sjis / reopen3: total time (10000 reps) was 2.122550 $ python3 bm.py test2.utf8 test2.utf8 / reopen0: total time (10000 reps) was 1.690255 test2.utf8 / reopen1: total time (10000 reps) was 1.996235 test2.utf8 / reopen2: total time (10000 reps) was 2.278798 test2.utf8 / reopen3: total time (10000 reps) was 2.727867 $ python3 bm.py test2.sjis test2.sjis / reopen0: total time (10000 reps) was 1.841388 test2.sjis / reopen1: total time (10000 reps) was 2.147142 test2.sjis / reopen2: total time (10000 reps) was 2.426701 test2.sjis / reopen3: total time (10000 reps) was 2.873278 ---------------------------------------------------------- Here is what happen when a test data file is piped into a program using the four methods above: $ cat test.utf8 | python3 stdin.py reopen0 read 102 characters $ cat test.utf8 | python3 stdin.py reopen1 got exception: [Errno 29] Illegal seek $ cat test.utf8 | python3 stdin.py reopen2 read 0 characters $ cat test.utf8 | python3 stdin.py reopen3 read 0 characters ---- [*1] Here is the get_encoding function used above. It is a toy simplified python source encoding line reader. Toy, in that is looks at only one line, doesn't consider a BOM, etc. It purpose was to allow me to sanity check the benefits of having a callable encoding parameter. def get_encoding (line): if isinstance (line, bytes): nlpos = line.index(b'\n') mo = ENC_PATTERN_B.search (line, 0, nlpos) if not mo: return None enc = mo.group(1).decode ('latin1') else: nlpos = line.index('\n') mo = ENC_PATTERN_S.search (line, 0, nlpos) if not mo: return None enc = mo.group(1) return enc

1 0

Isolated (?transactional) exec (?subinterpreter) calls
by anatoly techtonik 11 Jun '12

11 Jun '12

Hi, Having a lot of ideas is a curse, because I can barely follow up on them, but I try - I really read replies, just don't have enough energy to answer immediately (as it usually requires some time for research). Here is another one that ripes too long to become rotten: Make exec(code[, globals[, locals]]) calls consistent, optionally isolated from parent environment and transactional. Consistent: - it should not matter if the code is executed with command line interpreter or from exec(), code should not be modified to successfully run in exec if it successfully runs in intepreter session http://bugs.python.org/issue13557 http://bugs.python.org/issue14049 http://bugs.python.org/issue1167300 http://bugs.python.org/issue991196 Optionally isolated from parent environment: - a feature to execute user script in a snapshot of current environment and have a choice whenever to merge its modifications back to environment or not real user story - read system configuration settings, where optional detection rules are written in Python (Blender/SCons build scripts) - autodetection probes can affect environment while detection takes place and it can lead to more problems later (think of virtualenv on Python process level with defined data exchange protocol through globals/locals variables) Transactional: - well, if it is isolated - it is already transactional - an ability to discard results if an error or an exception inside exec() occurs - getting back to the state right before exec. -- anatoly t.

4 4

Re: [Python-ideas] Saving state in list/generator comprehension
by Andrew Carter 11 Jun '12

11 Jun '12

On Sun, Jun 10, 2012 at 7:01 PM, Steven D'Aprano <steve(a)pearwood.info>wrote: > Andrew Carter wrote: > >> Forgive me for any problems in this e-mail as I'm new to this mailing >> list. >> >> I thought it might be nice to be able to somehow save a state in >> list/generator comprehensions, >> a side effect of this (although not the intended goal) is it would make >> reduce feasible in a clean manner as the final result would just be the >> state. >> > > reduce already exists; in Python 2, it is a built-in available at all > times, in Python 3 it has been banished to the functools module. > > I think that it is was banished for good reason. I have found myself sometimes writing code that needs a reduce, but I feel that using the reduce function provided isn't very clear, especially if you see it and aren't familiar with the function. Admittedly writing a function that is a few short lines is possible, and what I end up doing, it just seems like there should be a more elegant way to do it than having a bunch of specialized functions. What is your use-case for this? "Saving state" is a means to an end. The > beauty of list comprehensions and generator expressions is that they are > intentionally quite simple and limited. If you need something more complex, > write a function or generator. Not everything has to be a (very-long and > unreadable) one-linear. > > reduce already exists, but if it didn't, you could write it quite easily. > Here's a version with optional starting value which yields the intermediate > results: > As I have mentioned above occasionally I want to turn a list into a single value by some repeated operation, but actually I think its more common that I want to map some operation over a list with dependencies of previous operation passed through. I think my most common use case, is I have a function that operates on a single value, and also has some state. Unfortunately leaving my example purposely vague, I was iterating over a list, and had what was effectively an environment variable (more of state) initially as a dynamic environment, so it was updated each time the function was called across the list comprehension. I then for other reasons wanted to use the environment type as a key for dictionary (which is a problem if its mutable), but that meant that the original list comprehension (which I felt was rather simple). Admittedly it didn't take me any time at all to write a simple function that did the list comprehension, but it still felt like a simple enough problem that it could be elegantly solved without resorting to the helper function. > import itertools > _MISSING = object() # sentinel value > > def foldl(func, iterable, start=_MISSING): > # foldr is left as an exercise :-) > if start is _MISSING: > it = iter(iterable) > else: > it = itertools.chain([start], iterable) > a = next(it) # raises if iterable is empty and start not given > try: > b = next(it) > except StopIteration: > yield a > return > a = func(a, b) > yield a > for b in it: > a = func(a, b) > yield a > > > Modifying this to return just the last value is easy, and in fact is > simpler than the above: > > def foldl(func, iterable, start=_MISSING): > if start is _MISSING: > it = iter(iterable) > else: > it = itertools.chain([start], iterable) > a = next(it) > for b in it: > a = func(a, b) > return a > > > > [...] > > Some of the pros of doing it this way is that because with/as are already >> keywords in python backwards compatibility shouldn't be an issue, >> > > That's not an argument in favour of your request. That's merely the lack > of one specific argument against it. There are an infinite number of things > which could be done that won't break backwards compatibility, but that > doesn't mean we should do them all. What positive arguments in favour of > your proposal do you have? What does your proposal allow us to do that we > can't already do, or at least do better? > > I feel like there is a need from personal experience of mapping with state in a short concise way. However it is quite possible that it is just me, and I need to think about the problem differently, or perhaps live with 4 line functions that are only used once. As for the backwards compatibility I think was getting ahead of myself, I feel the with/as solution is quite clunky, but I couldn't come up with a more elegant solution that operated in a similar vein to how python feels as a language. > > p.s. Is there a built-in way to get the last element from a generator >> (perhaps even with a default) a quick google search did not reveal one? >> > > The same as you would get the last element from any iterator, not just > generators: iterate over it as quickly as possible, keeping only the last > value seen. Because generator values are generated lazily as needed, > there's no direct way to skip to the last value, or get random access to > them. > > In pure Python: > > for x in iterator: > pass > > This may be faster: > > collections.deque(iterator, maxlen=1)[0] > > That's a neat solution, a little bit confusing at first glance, but still very neat, thanks! > Of course, both examples assume that the iterator or generator yields at > least one value, and is not infinite. > > > > -- > Steven > > ______________________________**_________________ > Python-ideas mailing list > Python-ideas(a)python.org > http://mail.python.org/**mailman/listinfo/python-ideas<http://mail.python.org/mailman/listinfo/python-ideas> >

2 1

Saving state in list/generator comprehension
by Andrew Carter 11 Jun '12

11 Jun '12

Forgive me for any problems in this e-mail as I'm new to this mailing list. I thought it might be nice to be able to somehow save a state in list/generator comprehensions, a side effect of this (although not the intended goal) is it would make reduce feasible in a clean manner as the final result would just be the state. One mechanism I can think of is to overload the with/as keyword for use inside of list/generator comprehensions, and using the previous result as state I believe the change to the grammar in python3k would be comp_iter : comp_for | comp_if | comp_with comp_with: 'with' testlist 'as' testlist So something in the form of [expr for i in iterable with initializer as accumulator] would resolve to something like result = [] accumulator = initializer for i in iterable: accumulator = expr result.append(accumulator) return result For instance reduce could be defined as (assuming all 3 arguments are required) reduce = lambda function, iterable, initializer : ([initializer] + [function(accumulator, i) for i in iterable with initializer as accumulator])[-1] Breaking this down, the "with initializer as accumulator" statement means that when the list comprehension begins accumulator=initializer, then after each iteration, accumulator = function(accumulator, i), so with the function f, list [i1,i2,i3,...], and initial value i0, the resulting list of "[function(accumulator, i) for i in iterable with initializer as accumulator]" would be [f(i0,i1), f(f(i0,i1),i2), f(f(f(i0,i1),i2),i3),...], or in left associative infix form with the f = "+" operator, [i0+i1,i0+i1+i2,i0+i1+i2+i3,...]. Consing (effectively) initializer to the beginning of the list ensures clean behavior for empty lists, and indexing [-1] gets the last element which is really the only element that matters. Consider a slightly more complex example of a Fibonacci generator, one might define it as follows, def fibs(): a, b = 1, 0 while True: a, b = b, a + b yield b Using the with statement, it would require two generator comprehensions fibs = lambda : (b for a,b in (b, a+b for i in itertools.cycle((None,)) with a,b = 0,1)) The inner generator comprehension (b, a+b for i in itertools.repeat(None) with a,b = 0,1) creates an infinite generator of tuples which are consecutive Fibonacci numbers, the outer list comprehension strips off the unneeded "state". Some of the pros of doing it this way is that because with/as are already keywords in python backwards compatibility shouldn't be an issue, but if one is just mapping with state then an extra list/generator comprehension block is needed to strip the state from the intermediate list. I apologize if similar ideas have already been discussed. -Andrew Carter p.s. Is there a built-in way to get the last element from a generator (perhaps even with a default) a quick google search did not reveal one?

2 1

BindError as a built-in TypeError subclass (on the margin of PEP 362 discussion)
by Jan Kaliszewski 11 Jun '12

11 Jun '12

Suggestion ========== I think that BindError proposed in PEP 362 could be a built-in TypeError subclass, raised whenever given arguments do not match a given callable: 1. while using Signature().bind(...) [as proposed in PEP 362], and also 2. while using inspect.getcallargs(...) and *also* 3. while doing *any* call. Rationale ========= The present behaviour (ad 2. and 3.), i.e. raising TypeError, makes it hard to differentiate call-argument-related errors from other TypeError occurrences. Raising BindError (or ArgumentError? the actual name is disputable of course), being a TypeError instance, instead -- would made easier implementing test suites, RPC mechanisms etc. Cheers. *j

6 8

Nudging beginners towards a more accurate mental model for loop else clauses
by Nick Coghlan 10 Jun '12

10 Jun '12

(context for python-ideas: my recently checked in changes to the tutorial, that added the final paragraph to http://docs.python.org/tutorial/controlflow.html#break-and-continue-stateme…) On Fri, Jun 8, 2012 at 5:29 PM, Stephen J. Turnbull <stephen(a)xemacs.org> wrote: > Note: reply-to set to python-ideas. > > Nick Coghlan writes: > > > The inaccuracies in the analogy are why this is in the tutorial, not the > > language reference. All 3 else clauses are really their own thing. > > Nick, for the purpose of the tutorial, actually there are 4 else > clauses: you need to distinguish *while* from *for*. It was much > easier for me to get confused about *for*. The only thing I'm trying to do with the tutorial update is to encourage beginners to be start thinking in terms of try/except/else when they first encounter for/break/else and while/break/else. That's it. Yes, ultimately once people fully understand how it works under the hood (including the loop-and-a-half construct for infinite while loops), they'll release it's actually closely related to conditionals as well, but anyone that places too much weight on the following obvious parallel is going to be confused for a long time. After all: if iterable: ... else: ... is *very* similar in appearance to: for x in iterable: ... else: ... I believe that parallel is 99% of the reason why people get confused about the meaning of the latter. The point of the tutorial update is to give readers a slight nudge towards thinking of the latter as: for x in iterable: ... except break: # Implicit in the semantics of loops pass else: ... Would it be worth adding the "except break:" clause to the language just to make it crystal clear what is actually going on? I don't think so, but it's still a handy way to explain the semantics while gently steering people away from linking for/else and if/else too closely. I actually agree all of the else clauses really *are* quite closely related (hence the consistent use of the same keyword), but the relationship is *not* the intuitively obvious one that comes to mind when you just look at the similarity in the concrete syntax specifically of for/else and if/else. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

15 31

Re: [Python-ideas] Replacing the standard IO streams (was Re: changing sys.stdout encoding)
by Rurpy 10 Jun '12

10 Jun '12

On 06/09/2012 08:26 PM, Nick Coghlan wrote: > Calling detach() on the standard streams is a bad idea - the > interpreter uses the originals internally, and calling detach() > breaks them. The documentation for sys.std* specifically describes using detach() on the standard streams: | To write or read binary data from/to the standard | streams, use the underlying binary buffer. and gives example code. The only caveat mentioned is that detach() "can raise AttributeError or io.UnsupportedOperation" if the stream has benn replaced with something that does not support detach().

1 0

Re: [Python-ideas] Nudging beginners towards a more accurate mental model for loop else clauses
by Zachary Ware 09 Jun '12

09 Jun '12

I've had a thought on this topic; how would it be to completely leave else out of the if, for, and while sections, then give else its own section explaining exactly how it works in each situation where it is applicable? I'd be happy to write up a sample later this evening if this thought isn't completely shot down :) As a side note, I didn't even know there was a while...else construct until I saw this discussion. I'd heard of for...else, but not with while.

1 0

Re: [Python-ideas] functools.partial
by Nick Coghlan 09 Jun '12

09 Jun '12

(Added list back to recipients) On Sat, Jun 9, 2012 at 10:58 AM, Devin Jeanpierre <jeanpierreda(a)gmail.com> wrote: > On Fri, Jun 8, 2012 at 8:17 PM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: >> Speed, complexity and backwards compatibility. With a layered API, users can >> choose whether they want to do early checks or not. If we build it in, you >> can't avoid it when you prefer the delayed error to checking the arguments >> twice. > > Then maybe the layered API belongs in the stdlib. What's the use of > base partial type, except as a micro-optimization? functools.partial will still be used to change the signature of a callable, the same as it has been ever since it was added. The layered API runs afoul of "not every three line function needs to be in the standard library". It's better to add the base API that is difficult for third parties to provide (in this case, inspect.signature and Signature objects) and let specific use cases emerge naturally over time, rather than trying to guess the *exact* patterns in advance. > On the other hand, I'm not so sure about your complexity argument. > It's hard to argue against "this isn't worth our time", if that's what > you're saying. But if you're saying it's too complicated now, > shouldn't Signature.bind help with that? I'm saying it makes *functools.partial* more complex, because we're asking it to do more. We would also be making it impossible to use without checking every signature twice. Current uses will be a mix of cases where the lack of early checking is annoying (but tolerable) and cases where it is undesirable. Adding the checking directly to the base API means we're assuming that early checking is desirable for *every* use case, and that's unlikely to be true. Most importantly though, if we leave the status quo in place for now, we can change our minds later if we still think it's a good idea. If we charge ahead and add early checking everywhere immediately, then we're quite likely to do more harm than good. We're in this for the long haul, and 2014 really isn't that far away in the context of programming language evolution. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0