Python-ideas May 2011

python-ideas@python.org

69 participants
31 discussions

Re: [Python-ideas] [Python-Dev] CPython optimization: storing reference counters outside of objects

by Sturla Molden

Den 23.05.2011 06:59, skrev "Martin v. Löwis": > > My expectation is that your approach would likely make the issues > worse in a multi-CPU setting. If you put multiple reference counters > into a contiguous block of memory, unrelated reference counters will > live in the same cache line. Consequentially, changing one reference > counter on one CPU will invalidate the cached reference counters of > that cache line on other CPU, making your problem a) actually worse. In a multi-threaded setting with concurrent thread accessing reference counts, this would certainly worsen the situation. In a single-threaded setting, this will likely be an improvement. CPython, however, has a GIL. Thus there is only one concurrently active thread with access to reference counts. On a thread switch in the interpreter, I think the performance result will depend on the nature of the Python code: If threads share a lot of objects, it could help to reduce the number of dirty cache lines. If threads mainly work on private objects, it would likely have the effect you predict. Which will dominate is hard to tell. Instead, we could use multiple heaps: Each Python thread could manage it's own heap for malloc and free (cf. HeapAlloc and HeapFree in Windows). Objects local to one thread only reside in the locally managed heap. When an object becomes shared by seveeral Python threads, it is moved from a local heap to the global heap of the process. Some objects, such as modules, would be stored directly onto the global heap. This way, objects only used by only one thread would never dirty cache lines used by other threads. This would also be a way to reduce the CPython dependency on the GIL. Only the global heap would need to be protected by the GIL, whereas the local heaps would not need any global synchronization. (I am setting follow-up to the Python Ideas list, it does not belong on Python dev.) Sturla Molden

9 years, 5 months

function defaults and an empty() builtin

by Jack Diederich

During a code review I got asked a question about a "pythonic" idiom I've been asked about before. The code was like this: def func(optional=None): if optional is None: optional = [] The question was why the optional value wasn't set to an empty list in the first place. The answer is that Really Bad Things can happen if someone actually goes and manipulates that empty list because all future callers will see the modified version. I don't think this defensive programming practice is yet passe - I can think of lots of unit tests that wouldn't trigger bad behavior. You would have to intentionally provoke it by adding some unit tests to be sure. What would make my life a little easier is a builtin container named "empty()" that emulates all builtin containers and raises an exception for any add/subtract manipulations. Something like: class empty(): def _bad_user(self, *args): raise ValueError("empty objects are empty") append = pop = __getitem__ = add = setdeafult = __ior__ = __iand__ = _bad_user def _empty(self): return [] items = keys = values = get = _empty return nothing when asked for something and raise a ValueError when any attempt is made to add/remove items. -Jack

9 years, 5 months

Filtered "for" loop with list-comprehension-like syntax

by Dan Baker

One common pattern I run across when parsing plain text data files is that I want to skip over blank lines when processing. If I wanted to build a list of all non-blank lines in the file, I could simply do: lines = [line for line in input_file if line.strip()] But as a loop, it almost invariably gets written as: for line in input_file: if not line.strip(): continue # do the real processing It seems odd that "for x in y if z" is allowed in comprehensions but not in a regular for loop. Why not let for x in y if z: do_stuff(x) be a shorthand for for x in y: if not z: continue do_stuff(x) Similarly, I occasionally have multiple sections that need to be handled differently. One way to write this is: for line in input_file: if is_section_delimiter(line): break do_stuff_1(line) for line in input_file: # this picks up where the last one left off if is_section_delimiter(line): break do_stuff_2(line) etc. (This is a little bit of a weird idiom with files since repeated iteration over them remembers where it left off, at least in 2.7.) It would be nice to have this shorthand for it: for line in input_file while not is_section_delimiter(line): do_stuff_1(line) for line in input_file while not is_section_delimiter(line): do_stuff_2(line) etc. This makes it more immediately clear (to me, at least) that it stops at the end of the section. This could also be added to comprehensions; it's somewhat tricky to emulate in comprehensions now. I think the easiest way to do the equivalent of [f(x) for x in y while z] with a comprehension is a = [(f(x) if z else None) for x in y] try: idx = a.index(None) except ValueError: # no None found pass else: # truncate before first None a = a[:idx] but even that fails if None is a potentially valid result of f(x) (or if you forget to use the try/except block and z was always True), and it processes the entire list even though it may throw out a sizable chunk of it immediately after. The only totally safe way I can think of to do it now is by unpacking it into a loop: a = [] for x in y: if not z: break a.append(f(x)) I think adding these would make such idioms a little more readable, but it might not be enough of a gain to justify a syntax addition. Thoughts? Dan Baker

9 years, 5 months

Type Metadata (and related ideas)

by andrew cooke

Hi, I just finished working on a project that plays around with ABCs and function annotations. The idea was to allow for more delcarative code by adding tools to describe Pytohn data in more detail. While I don't think the result is suitable for adding to Python (it's way too big a change, and it's not yet proven), the process of making something consistent involved working through a lot of ideas about "types in Python" that I recorded at http://www.acooke.org/pytyp.pdf Part of that paper (pages 8 and 9) describes some issues that caused particular problems, including: * The lack of annotations on type generators makes it hard to use annotations as a way of completely describing types. * There seems to be a missing ABC for __getitem__ (which unites lists, dicts and tuples). * As ever, mutability is complicated :o) If we had copy on write lists (which already exist), perhaps we could hash instances efficiently? (OK, this may be already discussed, but I had to mention it) * Given duck typing, shouldn't AttributeError be a TypeError (or vice versa?) * For this particular use-case, an __instancehook__ (which would work much like __subclasshook__) for ABCMeta would have been useful (as described in the paper, polymorphism in Python occurs at the instance level, so asking about the types of instances makes a surprising amount of sense, if done right). Anyway, apologies if some or all of this is old news or inapprorpiate. I just thought people here might find it interesting (you can do things like type check functions and use dynamic dispatch by type - all in a fairly pythonic way... (imho)) Cheers, Andrew PS The project home and more docs are at http://www.acooke.org/pytyp/ ; the code is at http://code.google.com/p/pytyp/ ; pypi page is http://pypi.python.org/pypi/pytyp

9 years, 5 months

PEP-3151 pattern-matching

by Devin Jeanpierre

Hello, PEP-3151 < http://www.python.org/dev/peps/pep-3151/ > mentions a really weird syntax for pattern-matching. I was hoping I could suggest an alternative that's both more concise, and possible to implement without doing something drastic like changing existing syntax or semantics. The PEP offers the pattern-matching syntax: > except IOError as e if e.errno == errno.ENOENT: ... I'd instead suggest something along the lines of > except io_error(errno.ENOENT): ... Implementing this is fairly straightforward, I've included it in the bottom of this email. Raising an exception becomes `raise io_error(errno.ENOENT)(msg)`. Some notational sugar can be implemented (for example, `raise io_error(errno.ENOENT, msg)` could also be made to work). I'm not fussed about the details. I personally prefer keeping the errnos as a big part of handling exceptions, they're common across OSes and involve less research / memorization for those that are already aware of errno. I guess what I'd like to see is the same exception hierarchy proposed by the PEP, but with the addition of allowing errnos to be specified by pattern-matching, so that errors not covered by the hierarchy, or more specific than the hierarchy, can be concisely caught. However, I'm not really well-versed in the pros and cons for all of this. Above all, I'd like for the pattern matching alternative to be a bit more reasonable. It doesn't have to be verbose and it doesn't have to involve new syntax. Apologies for any mistakes in the code, they are my own. Here's the code: # evil global state or something error_classes = {} def io_error(errno_, msg=None): # or something, you get the idea try: cls = error_classes[errno_] except LookupError: class IOErrorWithErrno(IOError): errno = errno_ cls = error_classes[errno_] = IOErrorWithErrno return error_classes[errno_] # example of usage import errno try: raise io_error(errno.ENOENT)("<Error message>") except io_error(errno.ENOENT): print("success") Thanks for your time! Devin Jeanpierre

9 years, 5 months

Re: [Python-ideas] [Python-Dev] Python 3.x and bytes

by Ethan Furman

Martin v. Löwis wrote [from python-dev]: > Immutable objects that compare equal should hash equal; > so we would also have to change the hashing of byte strings. Not sure > whether that, in turn, has undesirable consequences. I thought it was the other-way-round -- if they hash equal, they should compare equal? Or is this just for immutables? > In addition, equality should be transitive, so b'A' == 65.0. I'm not sure what you're getting at... we could certainly have step 2 check for a number instead of an int, and then step 3 could extract the one element, giving an int, and then let that int compare itself with the other number, whether it be int, float, fraction, what-have-you. ~Ethan~

9 years, 5 months

Rename python.exe to python3.exe on Windows

by Devin Jeanpierre

Hello, On most *nix systems, Python 3.x is available as the python3 executable, and Python 2.x as the 'python' executable. This lets both exist side-by-side and be usable from the command-line. The alternative (used by Arch), is to name Python 2.x 'python2', and 3.x 'python'. The Windows distribution of Python does neither, it names them both 'python.exe', meaning that you can't install and use both at once. Moreover, if you install Python 2.7 and then Python 3.2, the default handler for .py files is set to Python 3.2, and changing it to 2.7 is difficult because of a quirk in Eexplorer that forces you to choose between two non-distinguishable "python.exe"s. This is made much more difficult if in fact you installed five or so different Python versions. Also any automated tests using something like Cram that use python3 will not work, and any batch scripts that use python.exe will work differently depending on the host system. (It wouldn't be awful to get python-X.Y.exe executables, either). The downside of this is that any code that tries to use C:\Python3Y\python.exe breaks. Such code is probably broken anyway, there are multiple Ys around, and Python can be installed in My Documents or wherever. PEP 397 should relieve the issues with opening .py files, making some of this unnecessary with that change, as well. I'm guessing that it would also be appropriate to rename pythonw.exe to python3w.exe. I doubt that particular change matters at all, it's solely to do with opening .pyw files, and that should be handled by PEP 397. I'd appreciate any thoughts or comments you might have. Thanks for your time, Devin Jeanpierre

9 years, 5 months

Suggestion for Style Guide for Python Code PEP 8

by Rahul Amaram

Hi, I was wondering if the following programming recommendation would be added to the Style Guide for Python Code (PEP 8) page. The preferred way for checking if a key (k) exists in a dictionary (d) is "if k in d". This is faster than "if k in d.keys()" and this has superseded "d.has_key(k)" Cheers, Rahul.

9 years, 5 months

a few decorator recipes

by Mathias Panzenböck

A few decorator recipes that might be worthwhile to add to functools: A way to assign annotations to functions/classes: def annotations(**annots): def deco(obj): if hasattr(obj,'__annotations__'): obj.__annotations__.update(annots) else: obj.__annotations__ = annots return obj return deco _NONE = object() def getannot(obj, key, default=_NONE): if hasattr(obj, '__annotations__'): if default is _NONE: return obj.__annotations__[key] else: return obj.__annotations__.get(key, default) elif default is _NONE: raise KeyError(key) else: return default def setannot(obj, key, value): if hasattr(obj, '__annotations__'): obj.__annotations__[key] = value else: obj.__annotations__ = {key: value} Usage: >>> @annotations(foo='bar',egg='spam') >>> def foo(): >>> pass >>> >>> getannot(foo, 'egg') 'spam' A way to assign values to classes/functions (not of much use for classes, of course): def assign(**values): def deco(obj): for key in values: setattr(obj, key, values[key]) return obj return deco Usage: >>> @assign(bla='bleh',x=12) >>> def foo(): >>> pass >>> >>> foo.x 12

9 years, 5 months

triple-quoted strings and indendation

by Matthias Lehmann

Hi all, two times in one day I read about the problems of triple-quoted strings and indendation (one time on stackoverflow, one time one this list). Python is well known for its readability and its use of idendation to this end. But with triple-quoted strings, nice indendation is not possible without the need to post-process the resulting string. Problem ======= Most often, the desired result of >>> some_string = """Hello ... World.""" is simply Hello World instead of Hello World. Idea ===== What about the idea, to use a string-flag to indicate, that the triple-quoted string is to be trimmed. Like: >>> some_string = t"""Hello ... World.""" This would blend in with the 'u' and 'r' flags that already exist. The triple-quoted string is trimmed to remove all whitespace up to the column where the first line of the string started OR all common whitespace of the subsequent lines, if the subsequent lines start on a column before the first line. The second rule makes it possible to also write: >>> some_string = t"""Hello ... World.""" Pros ===== The advantages above textwrap.dedent are: 1) textwrap.dedent only removes whitespace common to ALL lines, so to achieve the desired result, one has to add an additional newline >>> some_string = """ ... Hello ... World.""" >>> result = textwrap.dedent(some_string)[1:] 2) Also, it does not work, if one actually does want some common whitespace before all lines: >>> some_string = """ ... Hello ... Wold.""" >>> result = textwrap.dedent(some_string)[1:] gives again Hello World which is not, what I wanted. But >>> some_string = t""" Hello ... World.""" would give Hello World. 3) And finally to quote a post from earlier today "I know about textwrap.dedent, but having to use a Python function call to code a literal has always made me uncomfortable." Problems ========= Common indendation style for triple-quoted string (as far as I know) is >>> foo = """blubber ... bla""" (align to first quote-char) but with this auto-trimming, it would look better to use >>> foo = t"""blubber ... bar""" (align to first char after triple quotes) The other stlye would still work, though - as long as one does not want to preserve leading whitespace. Maybe the t flag could also cause a leading and trailing newline to be removed, so that >>> foo = t""" ... Hallo ... World. ... """ would also result in Hello World. Maybe something like this has been proposed before - please be kind, if it is an old hat. Mat <javascript:void(0);>

9 years, 5 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas May 2011