Mailman 3 November 2012 - Python-ideas

Re: [Python-ideas] An alternate approach to async IO
by Sturla Molden 29 Nov '12

29 Nov '12

On 28.11.2012 16:49, Richard Oudkerk wrote: > You are assuming that GetQueuedCompletionStatus*() will never block > because of lack of work. GetQueuedCompletionStatusEx takes a time-out argument, it can be zero. Sturla

4 8

An error in multiprocessing on MacOSX?
by Sturla Molden 27 Nov '12

27 Nov '12

See this: http://mail.scipy.org/pipermail/numpy-discussion/2012-August/063593.html According to Apple enineers: """ For API outside of POSIX, including GCD and technologies like Accelerate, we do not support usage on both sides of a fork(). For this reason among others, use of fork() without exec is discouraged in general in processes that use layers above POSIX. """ Multiprocessing on OSX calls os.fork, but not os.exec. Thus, is multiprocessing errorneously implemented on Mac? Forking without calling exec means that only APIs inside POSIX can be used by the child process. For NumPy, it even affects functions like matrix multiplication when the accelerate framework is used for BLAS. Does multiprocessing needs a reimplementation on Mac to behave as it does on Windows? (Yes it would cripple it similarly to the crippled multiprocessing on Windows.) And what about Python itself? Is there any non-POSIX code in the interpreter? If it is, os.fork should be removed on Mac. Sturla

7 20

Uniquify attribute for lists
by Robrecht De Rouck 22 Nov '12

22 Nov '12

Hello, I just wanted to bring to your attention that an *attribute for removing duplicate elements* for lists would be a nice feature. *def uniquify(lis): seen = set() seen_add = seen.add return [ x for x in lis if x not in seen and not seen_add(x)]* * * The code is from this post<http://stackoverflow.com/questions/480214/how-do-you-remove-duplicates-from…>. Also check out this performance comparison<http://www.peterbe.com/plog/uniqifiers-benchmark> of uniquifying snippets. It would be useful to have a uniquify attribute for containers in general. Best regards, Robrecht * *

10 16

python_modules as default directory for dependencies in distutils
by Philipp Hagemeister 22 Nov '12

22 Nov '12

Currently, there are the following methods for installing dependencies: · Use the distribution's packaging (ignored here, all further points refer to setup.py/distutils) · Install them system-wide (default). This requires superuser rights and is basically guaranteed to conflict with some other application, especially if applications are choosy about the versions of Python packages they like. · Install them user-wide (--user), with pretty much the same downsides, plus that the application is now bound to the user installing it. · Manually invoke distutils with another path (error-prone and non-standard). · Give up and use virtualenv. While this works fine, it's a little bit heavy-handed to modify one's shell just to launch a potentially trivial application. Therefore, I'd like to suggest a new alternative location (--here = --root "./python_modules", intended to become default in Python 5), modeled after node's packaging system (http://goo.gl/dMRTC). The obvious advantage of installing all dependencies into a directory in the application root is that the application will work for every user, never conflict with any other application, and it is both easy to package dependencies (say, for an sftp-only rollout) and to delete all dependencies. Of course, this is not sufficient to replace virtualenv, but I believe a large majority of applications will (or at least should) run under any common python interpreter. Aside from the new flag in distutils, the site module should automatically look into ./python_modules , as if it were a second USER_SITE. In node, this scheme works so well that virtually nobody bothers to use system-wide installation, except when they want a binary to be available in the PATH for all users. This suggestion seems so obvious that it probably has been discussed before, but my google-fu is too weak to find it. If it has, I'd be glad to get a link to the old discussion. Thanks! - Philipp

5 11

thread safe dictionary initialisation from mappings: dict(non_dict_mapping)
by Anselm Kruis 20 Nov '12

20 Nov '12

Hello, I found the following annoying behaviour of dict(non_dict_mapping) and dict.update(non_dict_mapping), if non_dict_mapping implements collections.abc.Mapping but is not an instance of dict. In this case the implementations of dict() and dict.update() use PyDict_Merge(PyObject *a, PyObject *b, int override). The essential part of PyDict_Merge(a,b, override) is # update dict a with the content of mapping b. keys = b.keys() for key in keys: ... a[key] = b.__getitem__(key) This algorithm is susceptible to race conditions, if a second thread modifies the source mapping b between "b.keys()" and b.__getitem__(key): - If the second thread deletes an item from b, PyDict_Merge fails with a KeyError exception. - If the second thread inserts a new value and then modifies an existing value, a contains the modified value but not the new value. Of course the current behaviour is the best you can get with a "minimum mapping interface". My Goal ------- I want to be able to implement a mapping so that "dict(mapping)" works in an "atomic" way. Requirements for a solution --------------------------- - it does not modify the behaviour of existing code - the performance must be similar to the current implementation - simple to implement - sufficiently generic Proposal -------- My idea is to define a new optional method "__update_other__(setter)" for mappings. A minimal implementation of this method looks like: def __update_other__(self, setter): for key in self.keys(): setter(key, self[key]) Now it is possible to extend PyDict_Merge(PyObject *a, PyObject *b, int override) to check, if the source mapping b implements "__update_other__()". If it does, PyDict_Merge simply calls b.__update_other__. Otherwise it falls back to the current implementation using keys() and __getitem__(). Pseudo code for PyDict_Merge(PyObject *a, PyObject *b, int override) if hasattr(b, "__update__other__") and callable(b.__update_other__): if override: b.__update_other__(a.__setitem__) else: b.__update_other__(a.setdefault) return # old code follows keys = b.keys() for key in keys: a[key] = b.__getitem__(key) Example ------- A typical implementation of __update_other__ for a mapping with concurrent access looks like: def __update_other__(self, setter): # perform an appropriate locking with self._lock: for key in self._keys: setter(key, self._values[key]) Note 1: The __update_other__(self, setter) interface hides the nature of the caller. It is also possible to retrieve only the keys or only the values using suitable setter arguments. Note 2: In order to preserve the behaviour of existing code, the method __update_other__() must not be added to existing mapping implementations (collections.abc.Mapping, UserDict, ...). But adding support for __update_other__ to __init__() and update() of existing implementations is OK. Note 3: If we add add a test for __update_other__ to PyDict_Merge in front of the "PyDict_Check(b)" test this would also solve the issues described in http://bugs.python.org/issue1615701. A sub-class of dict could implement __update_other__ to prevent PyDict_Merge from accessing its underlying dict implementation. Note 4: Alternative approach Obviously it is possible to write "dict(non_dict_mapping.items())" (or even "dict(non_dict_mapping.iteritems())", if iteritems() is implemented in a suitable way), but constructing the list of items is an expensive operation. Additionally it is fairly common and pythonic to write "dict(mapping)" to get a copy of mapping. Anselm -- Dipl. Phys. Anselm Kruis science + computing ag Senior Solution Architect Ingolstädter Str. 22 email A.Kruis(a)science-computing.de 80807 München, Germany phone +49 89 356386 874 fax 737 www.science-computing.de -- Vorstandsvorsitzender/Chairman of the board of management: Gerd-Lothar Leonhart Vorstand/Board of Management: Dr. Bernd Finkbeiner, Michael Heinrichs, Dr. Arno Steitz, Dr. Ingrid Zech Vorsitzender des Aufsichtsrats/ Chairman of the Supervisory Board: Philippe Miltin Sitz/Registered Office: Tuebingen Registergericht/Registration Court: Stuttgart Registernummer/Commercial Register No.: HRB 382196

5 5

Re: [Python-ideas] CLI option for isolated mode
by Mike Meyer 19 Nov '12

19 Nov '12

Random832 <random832(a)fastmail.us> wrote: >On 11/15/2012 6:18 PM, Mike Meyer wrote: >> It's obviously true. The kernel (or shell, as the case may be) >> interprets the shebang line to find the executable an pick out the >> arguments to pass to the executable. The executable (Python) then >> interprets the arguments, without ever having seen the shebang line. >Right. It **interprets the arguments**. In this case, the single >argument is "-E -s", there's no reason it couldn't or shouldn't treat >that the same way as "-E" "-s". I've got three reasons to not do that: Special cases aren't special enough to break the rules. Errors should never pass silently. In the face of ambiguity, refuse the temptation to guess. If you knew you were handling a shebang line, it might be different. But if a user went out of their way to pass those arguments on the command line, you want to be very careful about undoing what they did. I think providing single-letter variants for options (which was why this was pointed out) is a better solution than guessing about what the user wanted to do to silently surpress an error to handle the special case of not being able to have multiple arguments on a shebang line (for some OS's?). -- Sent from my Android tablet with K-9 Mail. Please excuse my swyping.

2 1

With clauses for generator expressions
by Andrew Barnert 18 Nov '12

18 Nov '12

First, I realize that people regularly propose with expressions. This is not the same thing. The problem with the with statement is not that it can't be postfixed perl-style, or used in expressions. The problem is that it can't be used with generator expressions. Here's the suggestion: upperlines = (lines.upper() for line in file with open('foo', 'r') as file) This would be equivalent to: def foo(): with open('foo', 'r') as file: for line in file: yield line.upper() upperlines = foo() The motivation is that there is no way to write this properly using a with statement and a generator expression—in fact, the only way to get this right is with the generator function above. And almost nobody ever gets it right, even when you push them in the right direction (although occasionally they write a complex class that has the same effect). That's why we still have tons of code like this lying around: upperlines = (lines.upper() for line in open('foo', 'r')) Everyone knows that this only works with CPython, and isn't even quite right there, and yet people write it anyway, because there's no good alternative. The with clause is inherently part of the generator expression, because the scope has to be dynamic. The file has to be closed when iteration finishes, not when creating the generator finishes (or when the generator is cleaned up—which is closer, but still wrong). That's why a general-purpose "with expression" wouldn't actually help here; in fact, it would just make generator expressions with with clauses harder to parse. A with expression would have to be statically scoped to be general. For more details, see this: http://stupidpythonideas.blogspot.com/2012/11/with-clauses-for-generator-ex…

11 44

Speed up os.walk() 5x to 9x by using file attributes from FindFirst/NextFile() and readdir()
by Ben Hoyt 18 Nov '12

18 Nov '12

I've noticed that os.walk() is a lot slower than it needs to be because it does an os.stat() call for every file/directory. It does this because it uses listdir(), which doesn't return any file attributes. So instead of using the file information provided by FindFirstFile() / FindNextFile() and readdir() -- which listdir() calls -- os.walk() does a stat() on every file to see whether it's a directory or not. Both FindFirst/FindNext and readdir() give this information already. Using it would basically bring the number of system calls down from O(N) to O(log N). I've written a proof-of-concept (see [1] below) using ctypes and FindFirst/FindNext on Windows, showing that for sizeable directory trees it gives a 4x to 6x speedup -- so this is not a micro-optimization! I started trying the same thing with opendir/readdir on Linux, but don't have as much experience there, and wanted to get some feedback on the concept first. I assume it'd be a similar speedup by using d_type & DT_DIR from readdir(). The problem is even worse when you're calling os.walk() and then doing your own stat() on each file, for example, to get the total size of all files in a tree -- see [2]. It means it's calling stat() twice on every file, and I see about a 9x speedup in this scenario using the info FindFirst/Next provide. So there are a couple of things here: 1) The simplest thing to do would be to keep the APIs exactly the same, and get the ~5x speedup on os.walk() -- on Windows, unsure of the exact speedup on Linux. And on OS's where readdir() doesn't return "is directory" information, obviously it'd fall back to using the stat on each file. 2) We could significantly improve the API by adding a listdir_stat() or similar function, which would return a list of (filename, stat_result) tuples instead of just the names. That might be useful in its own right, but of course os.walk() could use it to speed itself up. Then of course it might be good to have walk_stat() which you could use to speed up the "summing sizes" cases. Other related improvements to the listdir/walk APIs that could be considered are: * Using the wildcard/glob that FindFirst/FindNext take to do filtering -- this would avoid fnmatch-ing and might speed up large operations, though I don't have numbers. Obviously we'd have to simulate this with fnmatch on non-Windows OSs, but this kind of filtering is something I've done with os.walk() many times, so just having the API option would be useful ("glob" keyword arg?). * Changing listdir() to yield instead of return a list (or adding yieldir?). This fits both the FindNext/readdir APIs, and would address issues like [3]. Anyway, cutting a long story short -- do folks think 1) is a good idea? What about some of the thoughts in 2)? In either case, what would be the best way to go further on this? Thanks, Ben. [1]: https://gist.github.com/4044946 [2]: http://stackoverflow.com/questions/2485719/very-quickly-getting-total-size-… [3]: http://stackoverflow.com/questions/4403598/list-files-in-a-folder-as-a-stre…

18 65

Fast pickle
by Serhiy Storchaka 17 Nov '12

17 Nov '12

I have an idea which can increase pickle/unpickle performance. This requires a change of format, so we need a new version of the protocol. May be it will be a 4 (PEP 3154) or 5. All items should be aligned to 4 bytes. It allow fast reading/writing of small integers, memcpy and UTF-8 codec should be faster on aligned data. In order not to waste space, a byte code should be combined with the data or the size of data. For integers: <code> <24-bit integer> <code> <24-bit size> <size-byte integer> <padding> <code> <56-bit size> <size-byte integer> <padding> For strings: <code> <8-bit size> <size-byte string> <padding> <code> <24-bit size> <size-byte string> <padding> <code> <56-bit size> <size-byte string> <padding> For collections: <code> <24-bit size> <item1> <item2> ... <item #size> <code> <56-bit size> <item1> <item2> ... <item #size> For references: <code> <24-bit index> <code> <56-bit index> For 1- and 2-byte integers, this can be expensive. We can add a special code for grouping. It will be even shorter than in the old protocols. <group code> <item code> <16-bit count> <integer1> <integer2> ... <integer #count> <padding> What do you think about this?

2 1

Support data: URLs in urllib
by Mathias Panzenböck 15 Nov '12

15 Nov '12

Sometimes it would be handy to read data:-urls just like any other url. While it is pretty easy to parse a data: url yourself I think it would be nice if urllib could do this for you. Example data url parser: >>> import base64 >>> import urllib.parse >>> >>> def read_data_url(url): >>> scheme, data = url.split(":") >>> assert scheme == "data", "unsupported scheme: "+scheme >>> mimetype, data = data.split(",") >>> if mimetype.endswith(";base64"): >>> return mimetype[:-7] or None, base64.b64decode(data.encode("UTF-8")) >>> else: >>> return mimetype or None, urllib.parse.unquote(data).encode("UTF-8") See also: http://tools.ietf.org/html/rfc2397 -panzi

3 12