Mailman 3 April 2011 - Python-ideas

Assignments in list/generator expressions
by Mathias Panzenböck April 12, 2011

April 12, 2011

I often have things like this: >>> ys = [f(x) for x in xs if f(x)] Obviously there is a redundant function call. You could rephrase that code to the following in order to avoid it: >>> ys = [y for y in (f(x) for x in xs) if y] But I think this is cumbersome and the extra generator overhead is unnecessary. So I propose this syntax: >>> ys = [f(x) as y for x in xs if y] It could be transformed to this: >>> ys = [] >>> for x in xs: >>&… [View More]

13 24

Re: [Python-ideas] Assignments in list/generator expressions
by Laura Creighton April 12, 2011

April 12, 2011

In a message of Sun, 10 Apr 2011 09:31:53 EDT, Eugene Toder writes: >> It isn't strangely missing at all, it's just hard: >> http://www.python.org/dev/peps/pep-3150/ > >Local definition in list comprehension is significantly simpler that >'given'. Semantically, 'with name = expr' can be treated as a more >readable form of 'for name in [expr]'. Implementation can use a simple >assignment. > >Eugene We'll have to disagree then. I think 'for name in [expr]' is a … [View More]

6 12

Re: [Python-ideas] Assignments in list/generator expressions
by Laura Creighton April 11, 2011

April 11, 2011

In a message of Sun, 10 Apr 2011 14:31:38 EDT, Eugene Toder writes: <snip> >It's well known that minor syntax details cause a lot of disagreement >(http://www.haskell.org/haskellwiki/Wadlers_Law), so that's not very >surprising. However, are you sure you're not confusing python's with >statement and 'with' word in my email, which is simply a placeholder >for appropriate keyword to mirror Haskell's 'let'? Or was this just an >occasion to complain about with statement … [View More]

3 2

AST Transformation Hooks for Domain Specific Languages
by Nick Coghlan April 10, 2011

April 10, 2011

(Oops, let's try that again with the correct destination address this time...) A few odds and ends from recent discussions finally clicked into something potentially interesting earlier this evening. Or possibly just something insane. I'm not quite decided on that point as yet (but leaning towards the latter). Anyway, without further ado, I present: AST Transformation Hooks for Domain Specific Languages ====================================================== Consider: # In some other module … [View More]ast.register_dsl("dsl.sql", dsl.sql.TransformAST) # In a module using that DSL import dsl.sql def lookup_address(name : dsl.sql.char, dob : dsl.sql.date) from dsl.sql: select address from people where name = {name} and dob = {dob} Suppose that the standard AST for the latter looked something like: DSL(syntax="dsl.sql", name='lookup_address', args=arguments( args=[arg(arg='name', annotation=<Normal AST for "dsl.sql.char">), arg(arg='dob', annotation=<Normal AST for "dsl.sql.date">)], vararg=None, varargannotation=None, kwonlyargs=[], kwarg=None, kwargannotation=None, defaults=[], kw_defaults=[]), body=[Expr(value=Str(s='select address\nfrom people\nwhere name = {name} and dob = {dob}'))], decorator_list=[], returns=None) (For those not familiar with the AST, the above is actually just the existing Function node with a "syntax" attribute added) At *compile* time (note, *not* function definition time), the registered AST transformation hook would be invoked and would replace that DSL node with "standard" AST nodes. For example, depending on the design of the DSL and its support code, the above example might be equivalent to: @dsl.sql.escape_and_validate_args def lookup_address(name: dsl.sql.char, dob: dsl.sql.date): args = dict(name=name, dob=dob) query = "select address\nfrom people\nwhere name = {name} and dob = {dob}" return dsl.sql.cursor(query, args) As a simpler example, consider something like: def f() from all_nonlocal: x += 1 y -= 2 That was translated at compile time into: def f(): nonlocal x, y x += 1 y -= 2 My first pass at a rough protocol for the AST transformers suggests they would only need two methods: get_cookie() - Magic cookie to add to PYC files containing instances of the DSL (allows recompilation to be forced if the DSL is updated) transform_AST(node) - a DSL() node is passed in, expected to return an AST containing no DSL nodes (SyntaxError if one is found) Attempts to use an unregistered DSL would trigger SyntaxError So there you are, that's the crazy idea. The stoning of the heretic may now commence :) Where this idea came from was the various discussions about "make statement" style constructs and a conversation I had with Eric Snow at Pycon about function definition time really being *too late* to do anything particularly interesting that couldn't already be handled better in other ways. Some tricks Dave Malcolm had done to support Python level manipulation of the AST during compilation also played a big part, as did Eugene Toder's efforts to add an AST optimisation step to the compilation process. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia [View Less]

7 16

Developer wish list
by geremy condra April 10, 2011

April 10, 2011

Earlier this year there was some discussion[0] about putting up a page on the wiki where developers could list the feature proposals they most wanted and most hated for the benefit of those posting to python-ideas. It's taken me a while to get around to it, but I've put up a skeleton for the page at [1] and would love it if some of you guys would take a look, let me know what you like/don't like, and maybe even post a few of your pet projects or pet peeves. Thanks for your time and effort, … [View More]

1 0

Re: [Python-ideas] [Python-Dev] AST Transformation Hooks for Domain Specific Languages
by Nick Coghlan April 9, 2011

April 9, 2011

(Fixed list to be python-ideas) On Sat, Apr 9, 2011 at 2:50 AM, David Malcolm <dmalcolm(a)redhat.com> wrote: > On Fri, 2011-04-08 at 21:29 +1000, Nick Coghlan wrote: >> AST Transformation Hooks for Domain Specific Languages >> ====================================================== > > This reminds me a lot of Mython: > http://mython.org/ > If you haven't seen it, it's well worth a look. Ah, very interesting - indeed, compile-time metaprogramming is … [View More]definitely what this idea is about (with all of the tremendous power and potentially major liabilities for readability and debugging that entails). > My favourite use case for this kind of thing is having the ability to > embed shell pipelines into Python code, by transforming bash-style > syntax into subprocess calls (it's almost possible to do all this in > regular Python by overloading the | and > operators, but not quite). > >> Consider: >> >> # In some other module >> ast.register_dsl("dsl.sql", dsl.sql.TransformAST) > > Where is this registered? Do you have to import this "other module" > before importing the module using "dsl.sql" ? It sounds like this is > global state for the interpreter. Yep - the registration would be with the compiler itself. Technically a "flat" namespace, but allowing dots in the names means it can mirror the general Python module namespace. An alternative would be to use the module namespace rather than a dedicated one, in which case registration might not be necessary. However, requiring an explicit registration step to support DSLs doesn't really bother me, even it means they can't be used as standalone scripts, but instead must be executed via a module that registers the DSL. With runpy.run_module and runpy.run_path available, supporting execution of other scripts isn't a great burden. >> # In a module using that DSL > > How is this usage expressed? via the following line? > >> import dsl.sql > > I see the "import dsl.sql" here, but surely you have to somehow process > the "import" in order to handle the rest of the parsing. No, it's expressed by the "from dsl.sql" at the end of the line: def lookup_address(name : dsl.sql.char, dob : dsl.sql.date) from dsl.sql: The import is included in the example, simply because that was a pattern I used in my sample expansion. As Jon pointed out in his reply, it's basically a very similar idea to Mython's "quote" block, but leveraging off the existing Function node to pick up other fun tricks like arguments, decorators and annotations rather than defining a completely separate statement. > Where and how would the bytes of the file usage the DSL get converted to > an in-memory tree representation? > > IIRC, manipulating AST nodes in CPython requires some care: the parser > has its own allocator (PyArena), and the entities it allocates have a > shared lifetime that ends when PyArena_Free occurs. They survive if you use PyCF_ONLY_AST (or otherwise get hold of the AST from Python). The arenas are just Python lists and returning the AST for use by Python code bumps the reference count of the head node. >> So there you are, that's the crazy idea. The stoning of the heretic >> may now commence :) > > Or, less violently, take it to python-ideas? (though I'm not subscribed > there, fwiw, make of that what you will) Yeah, sorry about that - the posting to python-dev was a mistake due to not properly checking the address auto-complete in Gmail. > One "exciting" aspect of this is that if someone changes the DSL file, > the meaning of all of your code changes from under you. This may or may > not be a sane approach to software development :) > (I also worry what this means e.g. for people writing text editors, > syntax highlighters, etc; insert usual Alan Perlis quote about syntactic > sugar causing cancer of the semicolon) Yeah, there's a way to experiment with this that's much friendlier to those systems: require that the body of the statement *also* be standard Python code. Support for non-Python syntax would then be handled by explicitly embedding it in a string (potentially even the docstring). The advantage this would have over existing string parsing techniques is that the contents would be checked at compile time rather than runtime, just as Mython does. > Also, insert usual comments about the need to think about how > non-CPython implementations of Python would go about implementing such > ideas. This is also AST-time, so the back end shouldn't make a big difference. >> Where this idea came from was the various discussions about "make >> statement" style constructs and a conversation I had with Eric Snow at >> Pycon about function definition time really being *too late* to do >> anything particularly interesting that couldn't already be handled >> better in other ways. Some tricks Dave Malcolm had done to support >> Python level manipulation of the AST during compilation also played a >> big part, as did Eugene Toder's efforts to add an AST optimisation >> step to the compilation process. > > Like I said earlier, have a look at Mython Indeed, in a lot of ways, this idea is really just an alternative syntax proposal for Mython "quote" statements, along with the concept of integrating it into the main compiler. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia [View Less]

1 0

Bad code exceptions (was PEP-3151 pattern-matching)
by Mike Graham April 8, 2011

April 8, 2011

On Fri, Apr 8, 2011 at 1:11 PM, Guido van Rossum <guido(a)python.org> wrote: > ... > > - Lastly, there is a bunch of standard exception types that are > usually caused by bad code (as opposed to bad data or an environmental > problem). This would include NameError, TypeError, AttributeError, but > not KeyError, IndexError, ValueError. Perhaps it makes sense to > introduce a common base class for these "bad code" exceptions? Unlike > the other exception types, I … [View More]

3 5

Re: [Python-ideas] [Python-Ideas] AST Transformation Hooks for Domain Specific Languages
by Eric Snow April 8, 2011

April 8, 2011

On Fri, Apr 8, 2011 at 5:29 AM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > A few odds and ends from recent discussions finally clicked into > something potentially interesting earlier this evening. Or possibly > just something insane. I'm not quite decided on that point as yet (but > leaning towards the latter). > > Anyway, without further ado, I present: > > AST Transformation Hooks for Domain Specific Languages > =============================================… [View More]========= > > Consider: > > # In some other module > ast.register_dsl("dsl.sql", dsl.sql.TransformAST) > > # In a module using that DSL > import dsl.sql > def lookup_address(name : dsl.sql.char, dob : dsl.sql.date) from dsl.sql: > select address > from people > where name = {name} and dob = {dob} > > I like how you moved the from to after the parameter list. It makes it less complicated. Annotations ( -> int: ) will still go after that, right? > Suppose that the standard AST for the latter looked something like: > > DSL(syntax="dsl.sql", > name='lookup_address', > args=arguments( > args=[arg(arg='name', > annotation=<Normal AST for "dsl.sql.char">), > arg(arg='dob', > annotation=<Normal AST for "dsl.sql.date">)], > vararg=None, varargannotation=None, > kwonlyargs=[], kwarg=None, kwargannotation=None, > defaults=[], kw_defaults=[]), > body=[Expr(value=Str(s='select address\nfrom people\nwhere > name = {name} and dob = {dob}'))], > decorator_list=[], > returns=None) > > (For those not familiar with the AST, the above is actually just the > existing Function node with a "syntax" attribute added) > > At *compile* time (note, *not* function definition time), the > registered AST transformation hook would be invoked and would replace > that DSL node with "standard" AST nodes. > > So the AST transform would not necessarily return a Function node... > For example, depending on the design of the DSL and its support code, > the above example might be equivalent to: > > @dsl.sql.escape_and_validate_args > def lookup_address(name: dsl.sql.char, dob: dsl.sql.date): > args = dict(name=name, dob=dob) > query = "select address\nfrom people\nwhere name = {name} and > dob = {dob}" > return dsl.sql.cursor(query, args) > > > As a simpler example, consider something like: > > def f() from all_nonlocal: > x += 1 > y -= 2 > > That was translated at compile time into: > > def f(): > nonlocal x, y > x += 1 > y -= 2 > > My first pass at a rough protocol for the AST transformers suggests > they would only need two methods: > > get_cookie() - Magic cookie to add to PYC files containing instances > of the DSL (allows recompilation to be forced if the DSL is updated) > transform_AST(node) - a DSL() node is passed in, expected to return > an AST containing no DSL nodes (SyntaxError if one is found) > > Attempts to use an unregistered DSL would trigger SyntaxError > > So there you are, that's the crazy idea. The stoning of the heretic > may now commence :) > > Where this idea came from was the various discussions about "make > statement" style constructs and a conversation I had with Eric Snow at > Pycon about function definition time really being *too late* to do > anything particularly interesting that couldn't already be handled > better in other ways. Some tricks Dave Malcolm had done to support > Python level manipulation of the AST during compilation also played a > big part, as did Eugene Toder's efforts to add an AST optimisation > step to the compilation process. > > I Like this idea better than the def-from with builders. Since it is compile-time, it certainly addresses some of the concerns that Guido brought up, like run-time dynamic code generation and pyc files. You simply have to play within the confines of the AST, which seems like an effective constraint. Like Michael said, this would be great for trying out new language features. And like Eugene said, it may be worth doing in an import hook. I've actually been working on such a hook since the feedback I got for those three ideas last week. I doubt I will have anything right away, but I'll let you know when I get there. Regardless, I still like the idea of a syntax that allows for the same freedom to define new stuff. I wonder though if it would be subject to extensive abuse that would make code harder to understand/use/debug. I don't know that there are any other language features that are extensively used that have his effect. _maybe_ metaclasses, decorators, and the whole dot-lookup mechanism. However, you can wrap your head around those with a little explanation. I guess this would be similar. It's good that the non-python is contained within the def body. I'm just thinking about what Raymond said last week about how all the meta-programming stuff makes debugging harder. I tend to agree. I think of the challenge of tracing through a problem I had in sqlalchemy a couple of years ago that involved their use of metaclasses. Anyway, love the idea. -eric > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev(a)python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ericsnowcurrently%40gmail… > [View Less]

1 0

Extending properties to sequence-like behavior
by Dan Baker April 7, 2011

April 7, 2011

I know the property() built-in allows you to override the behavior of attribute access in a class to call arbitrary functions where the syntax looks like a basic assign or lookup. This is handy when you're importing an external API and want to make it look more Pythonic. Is there a simple way to extend this so that item lookup or assignment can be similarly overridden? For instance, if the object I'm importing (say, a drop-down box in a GUI toolkit) has methods called GetAllItems, SetAllItems, … [View More]

5 6

A meaningful `if counter:`
by cool-RR April 6, 2011

April 6, 2011

Hello folks, I noticed today that bool(Counter({'a': 0'})) is True. Is this wise? I want to be able to do: if my_counter: whatever To check whether my counter has any elements. Currently this seems to be impossible because of this behavior. Will we have to keep this weird behavior because of backwards compatibility? If so, perhaps `.elements` could be turned into a smart object so we could at least do `if my_counter.elements():` and get the expected result. To people who use `… [View More]

4 12

Improving the expressivity of function annotations
by Masklinn April 5, 2011

April 5, 2011

While the PEP 3107 annotations are pretty nice, and an interesting improvements re third-party static analysis of Python code, I think they are insufficient in three areas which makes them… insufficient. And even if third-parties solved these issues independently, it likely would not reach criticality. type parameters (generics) ========================== Python simply has no syntax (at the moment) to express the idea of generics and generic containers. The simplest way (and one I find … [View More]syntactically clean, YMMV) would be to define ``__getitem__`` on ``type`` instances (with e.g. ``list[int]`` yielding a list with some kind of ``generics`` attribute), but that is of course unavailable to third-party libraries as they should not override builtins. Plus it would have to be added on ``type`` itself to ensure all types have this property (and I am not sure how C types are defined in relation to ``type``). The possibility to perform this call would probably need to be opt-in or opt-out, as generics don't make sense for all types (or maybe all types could accept an empty generic spec e.g. ``int[]``? though I'm not sure that is even valid python syntax) Type parameters greatly increase the expressivity of a type specification and make deep type checks possible, creating much static safety by avoiding implicit or explicit (unsafe) type casts. sum types ========= It is not uncommon for Python functions and methods to take or return a value from a set of types rather than from a type itself. That is also (I would guess) one of the reasons why ``isinstance`` can take a tuple of types rather than being limited to a single type (as with Javascript's or Java's ``instanceof``). As previously, it would probably be easy to alter ``type`` to syntactically support this construct: defining ``__or__`` on ``type`` as yielding some kind of TypeSet (or Types) instance, which would be a set of all OR'ed types (with the right subclass hooks set up so ``isinstance`` works correctly). It would result in something like: Sum = (A | B | C) assert issubclass(A, Sum) assert issubclass(B, Sum) assert issubclass(C, Sum) Structural types ================ The third issue is also the biggest by far. A small portion of it is resolved by abcs (especially if tools include special support for ABCs), but not all of it by a long shot: statically defining and checking "duck types". abcs check for structure, so they are a convenient label for *some* "duck" types but their meaning isn't even well defined (e.g. does the ``item: Mapping`` signature mean ``item`` implements the abstract methods of Mapping, or does it have to *also* implement the mixin methods, even if Mapping was not actually mixed in it?) Anyway this is where "structural types" come in: defining a type not by its name but by its shape (a set of methods and properties, and their signatures). Again, this can be handled by defining (enough) abcs, but it soon becomes tiresome as ABCs are still pretty verbose to write and I don't doubt there would soon be about 5 million versions of ``Readable`` (requirement of a single no-args method ``read`` producing bytes). As a result, the ability to define a type as a set of methods (and their signatures) without having to give it an explicit name would — I think — be great. On the other hand, this is *the* case for which I have no syntactical idea, definitely not a simple one, by simply augmenting existing types with new properties (without introducing actual new syntax). The OCaml syntax ``< method: (*input) -> output; method2: (*input) -> output; property: output >`` definitely does not fit. Your thoughts? [View Less]

10 27

Adding a half-float (16-bit) type to PEP 3118 (and possibly the struct module?)
by Eli Stevens (Gmail) April 5, 2011

April 5, 2011

Hello, Numpy 1.6.0 adds support for a half-float (16-bit) data type, but cannot currently export a buffer interface to the data, since the closest type that PEP 3118 supports is an unsigned short ('H'). This makes working with the data from outside numpy (for example, from Cython) difficult, since even if numpy were to expose a buffer interface to the data, it's unclear that the data needs special treatment to interpret correctly (numpy does this with bit shifting functions to convert it to a … [View More]

10 27

Dict view on list of two-tuples
by Ian Bicking April 5, 2011

April 5, 2011

It would be nice if the stdlib had a data structure that represented a dict view on a list of key/value tuples. This has come up a bit when thinking about common web request/response objects, as they all have an object like this but they all have slightly different APIs (and sometimes slightly different notions of what they want to do). For instance: d = multidict([('a', '1'), ('b', '2'), ('a', '3')]) There are some open questions: d['a'] -- '1' or '3'? d.items() -- could be [('a', '1'), ('… [View More]

3 5