Mailman 3 February 2012 - Python-ideas

Generators' and iterators' __add__ method
by Yuval Greenfield Feb. 16, 2012

Feb. 16, 2012

Wouldn't it be nice to add generators and iterators like we can do with lists? def f(): yield 1 yield 2 yield 3 def g(): yield 4 yield 5 # today for item in itertools.chain(f(), g()): print(item) # proposal for item in f() + g(): print(item) What do you guys think? Yuval Greenfield

6 8

automation of __repr__/__str__ for all the common simple cases
by Ronny Pfannschmidt Feb. 16, 2012

Feb. 16, 2012

Hi, in my experience for many cases, __repr__ and __str__ can be unconditionally be represented as simple string formatting operation, so i would propose to add a extension to support simply declaring them in the form of newstyle format strings a basic implementation for __repr__ could look like: class SelfFormatter(string.Formatter): def __init__(self, obj): self.__obj = obj string.Formatter.__init__(self) def get_value(self, key, args, kwargs): if isinstance(key, str) and hasattr(self.__obj, key): return getattr(self.__obj, key) return Formatter.get_value(self, key, args, kwargs) class SimpleReprMixing(object): _repr_ = '<{__class__.__name__} at 0x{__id__!x}>' def __repr__(self): formatter = SelfFormatter(self) return formatter.vformat(self._repr_, (), {'__id__':id(self)}) -- Ronny Pfannschmidt

7 12

Unicode surrogateescape [was: Re: Python 3000 TIOBE -3%]
by Jim Jewett Feb. 15, 2012

Feb. 15, 2012

On Mon, Feb 13, 2012 at 12:12 AM, Stephen J. Turnbull <stephen(a)xemacs.org> wrote: > Paul Moore writes: > > I'm now 100% convinced that > > encoding="ascii",errors="surrogateescape" is the way to say this in > > code. > That may also be a good universal default for Python 3, as it will > pass through non-ASCII text unchanged, while raising an error if the > program tries to manipulate it (or hand it to a module that > validates). (encoding='latin-1' definitely is not a good default.) > But I'm not sure of that, and the current approach of using the > preferred system encoding is probably better. The preferred system encoding is indeed better than universal ASCII. But is there a good reason not to change the default errorhandler to errors="surrogateescape"? errors="strict" is already well-documented, and the sort of people most eager to reject (rather than ignore) bad data also tend to be explicit about their use of defaults. And if the barrier is only backwards-compatibility, is there any reason not to at least recommend a recipe of errors="surrogateescape" for cases where you expect ASCII, but want to round-trip other data just in case? -jJ

3 2

Re: [Python-ideas] Py3 unicode impositions
by Jim Jewett Feb. 15, 2012

Feb. 15, 2012

On Fri, Feb 10, 2012 at 3:41 AM, Stephen J. Turnbull <stephen(a)xemacs.org> wrote: > Terry Reedy writes: > > > > In python 2 there was no such a strong imposition [of Unicode > > > awareness on users]. > > Nor is there in 3.x. > Sorry, Terry, but you're basically wrong here. True, if one sticks to > pure ASCII, there's no difference to notice, but that's just not > possible for people who live outside of the U.S., or who share text > with people outside of the U.S. They need currency symbols, they > have friends whose names have little dots on them. Every single > one of those is a backtrace waiting to happen. A backtrace on > f = open('text-file.txt') > for line in f: pass > is an imposition. That doesn't happen in 2.x (for the wrong reasons, > but it's very convenient 95% of the time). I may be missing something, but as best I can tell (1) That uses an implicit encoding of None. (2) encoding=None is documented as being platform-dependent. Are you saying that some (many? all?) platforms make a bad choice there? Does that only happen when sys.getdefaultencoding() != sys.getfilesystemencoding(), or when one of them gives bad information? (FWIW, on a mostly ASCII windows machine, the default is utf-8 but the filesystem encoding is mbcs, so merely being different doesn't always provoke problems.) Would it cause problems to make the default be whatever locale returns, or whatever it returns the first time open is called? -jJ

19 36

map iterator
by Edward Lesmes Feb. 14, 2012

Feb. 14, 2012

An iterator version of map should be available for large sets of data. -- Edward Lesmes

4 3

Re: [Python-ideas] Py3 unicode impositions
by Dirkjan Ochtman Feb. 14, 2012

Feb. 14, 2012

(adding back the list...) On Tue, Feb 14, 2012 at 21:55, Jim Jewett <jimjjewett(a)gmail.com> wrote: > As useful as it could be, it needs to clearly be an "external > standard", and should probably stay external. I don't want a > situation where "Hey, this file was detected differently in Py3.4 and > Py3.5" is a regression bug, or there will never be room for > improvements. Well, there have been *very* limited functional changes in the Mozilla tree since about 2008, so I don't think there would be a great many changes. And there's a large test suite to make sure regressions are unlikely. I still think the benefit for the simple cases is tremendous: a simple one-liner that finds the correct encoding in many of the cases. There's also very little API surface. Also, there is an actual 3.x port, I have it installed... should be in the same tarball. Cheers, Dirkjan

1 0

Unicode surrogateescape [was: Re: Python 3000 TIOBE -3%]
by Jim Jewett Feb. 14, 2012

Feb. 14, 2012

On Mon, Feb 13, 2012 at 12:16 AM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > Really, Python 3 forces programmers ... > to make the choice between the 4 possible options for > processing ASCII-compatible encodings: > 1. Process them as binary data. [Code smell from lying; lots of pain from mismatch with external libraries.] > 2. Process them as "latin-1". [Code smell from lying; non-ASCII often turns to gibberish.] > 3. Process them as "ascii+surrogateescape". This is the *right* > answer if you plan solely to manipulate the text and then write it back > out in the same encoding as was originally received. [Note that the original "encoding" may well be internally inconsistent; I've often seen that in log files.] > You will get errors if you try to write a string with escaped > characters out to a non-ascii channel or an ascii channel > without surrogateescape enabled. ... (e.g. sys.stdout) Is there any reason not to enable surrogate escape by default? At least on the console/terminal? I can see an argument for replace or xmlcharreplace or something more complicated, but ... if I'm sending output to myself, I would rather see it (possibly with a mark indicating where it was corrupted) than to get my program aborted (strict) and *not* be told what data caused the problem. > 4. Get a third party encoding guessing library and use that instead of > waving away the problem of ASCII-incompatible encodings. And I do think this needs to stay 3rd-party; domain information matters, and n-gram guessing should not be subject to stability guarantees. -jJ

1 0

Re: [Python-ideas] Python 3000 TIOBE -3%
by Mike Meyer Feb. 13, 2012

Feb. 13, 2012

On Sun, 12 Feb 2012 18:36:40 -0800 "C. Titus Brown" <ctb(a)msu.edu> wrote: > "All of them except subprocess, on some platforms" is the answer, AFAIK. Which > is kind of the point. Do you have any documentation to back this up? For instance, The collections and random module are both known to have code in them that isn't thread safe. For the random module, you can check the docstring: Help on method gauss in module random: gauss(self, mu, sigma) method of random.Random instance Gaussian distribution. mu is the mean, and sigma is the standard deviation. This is slightly faster than the normalvariate() function. Not thread-safe without a lock around calls. For the collections module, I quote the functools module: lock = Lock() # needed because ordereddicts aren't threadsafe The argparse and pprint modules both use ordereddicts without either locking them providing an explanation as to why they don't need to, which makes both of them suspect as well. Given those cases, I'm not willing to trust a simple assertion that a module is thread-safe, unless it's from the author or a primary maintainer of the module, or someone who's actually audited the module in question for thread safety. <mike -- Mike Meyer <mwm(a)mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

3 2

multiprocessing IPC
by Sturla Molden Feb. 13, 2012

Feb. 13, 2012

Den 10.02.2012 22:15, skrev Mike Meyer: > In what way does the mmap module fail to provide your binary file > interface? <mike The short answer is that BSD mmap creates an anonymous kernel object. When working with multiprocessing for a while, one comes to the conclusion that we really need named kernel objects. Here are two simple fail cases for anonymous kernel objects: - Process A spawns/forks process B. - Process B creates an object, one of the attributes is a lock. - Fail: This object cannot be communicated back to process A. B inherits from A, A does not inherit from B. - Process A spawns/forks a process pool. - Process A creates an object, one of the attributes is a lock. - Fail: This object cannot be communicated to the pool. They do not inherit new handles from A after they are started. All of multiprocessing's IPC classes suffer from this! Solution: Use named kernel objects for IPC, pickle the name. I made a shared memory array for NumPy that workes like this -- implemented by memory mapping from the paging file on Windows, System V IPC on Linux. Underneath is an extension class that allocates a shared memory buffer. When pickled it encodes the kernel name, not its content, and unpickling opens the object given its name. There is another drawback too: The speed of pickle. For example, sharing NumPy arrays with pickle is not faster with shared memory. The overhead from pickle completely dominate the time needed for IPC . That is why I want a type specialized or a binary channel. Making this from the named shared memory class I already have is a no-brainer. So that is my other objection against multiprocessing. That is: 1. Object sharing by handle inheritance fails when kernel objects must be passed back to the parent process or to a process pool. We need IPC objects that have a name in the kernel, so they can be created and shared in retrospect. 2. IPC with multiprocessing is too slow due to pickle. We need something that does not use pickle. (E.g. shared memory, but not by means of mmap.) It might be that the pipe or socket in multiprocessing will do this (I have not looked at it carefully enough), but they still don't have Proof of concept: http://dl.dropbox.com/u/12464039/sharedmem-feb12-2009.zip Dependency on Cython and NumPy should probably be removed, never mind that. Important part is this: sharedmemory_sysv.pyx (Linux) sharedmemory_win.pyx and ntqueryobject.c (Windows) Finally, I'd like to say that I think Python's standard lib should support high-performance asynchronous I/O for concurrency. That is not poll/select (on Windows it does not even work properly). Rather, I want IOCP on Windows, epoll on Linux, and kqueue on Mac. (Yes I know about twisted.) There should also be a requirement that it works with multiprocessing. E.g. if we open a process pool, the processes should be able to use the same IOCP. In other words some highly scalable asynchronous I/O that works with multiprocessing. So ... As far as I am concerned, the only thing worth keeping in multipricessing is multiprocessing.Process and multiprocessing.Pool. The rest doesn't do what we want. Sturla

5 16

The concurrency discussion is off-topic!
by Mike Meyer Feb. 13, 2012

Feb. 13, 2012

Please take the concurrency discussion to: http://mail.python.org/mailman/listinfo/concurrency-sig -- Mike Meyer <mwm(a)mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

9 19