Mailman 3 May 2013 - Python-ideas

Macros for Python
by Haoyi Li May 14, 2013

May 14, 2013

I thought this may be of interest to some people on this list, even if not strictly an "idea". I'm working on MacroPy <https://github.com/lihaoyi/macropy>, a little pure-python library that allows user-defined AST rewrites as part of the import process (using PEP 302). In short, it makes mucking around with Python's semantics so easy as to be almost trivial: you write a function that takes an AST and returns an AST, register it as a macro, and you're off to the races. To give a sense of it, I just finished implementing Scala/Groovy style anonymous lambdas: map(f%(_ + 1), [1, 2, 3])#[2, 3, 4] reduce(f%(_ + _), [1, 2, 3])#6 ...which took about half an hour and 30 lines of code, start to finish. We're currently working on implementing destructuring-pattern-matching on objects (i.e. like in Haskell/Scala) and a clone of .NET's LINQ to SQL. It's still very much a work in progress, but we have a list of pretty cool macros already done, which shows off what you can do with it. If anyone else was thinking about messing around with the semantics of the Python language but was too scared to jump into the CPython internals, this offers a somewhat easier path. Hope this was interesting to somebody! -Haoyi

7 15

Re: [Python-ideas] Syntax for easy binding __name__, __module__, __qualname__ to arbitrary objects
by Markus Unterwaditzer May 14, 2013

May 14, 2013

(For some reason my mail software decided not to cc the list) -------- Original Message -------- From: Markus Unterwaditzer <markus(a)unterwaditzer.net> Sent: Tue May 14 07:31:33 CEST 2013 To: Greg Ewing <greg.ewing(a)canterbury.ac.nz> Subject: Re: [Python-ideas] Syntax for easy binding __name__, __module__, __qualname__ to arbitrary objects Greg Ewing <greg.ewing(a)canterbury.ac.nz> wrote: >I'm not entirely happy with the current proposal: > > def name = expr > >because it doesn't fully entwine. The expr can be a >constructor, but doesn't have to be, and even when it >is, the construction occurs separately from the >assignment. Also, it looks like an ordinary assignment >with 'def' stuck in front, which, as Guido points >out, seems somewhat random. I don't agree with that, i think it's good the proposed syntax looks similar to an assignment, as the feature is clearly related to assignments. > >I'd like to propose something a bit different: > > > def name as expr(arg, ...) > >which would expand to something like > > name = expr(arg, ..., __name__ = 'name', __module__ = 'module') To me that implies that the __init__ method of a class has to implement explicit support for this feature, and that there's no way to make a standard implementation for object. I like the proposed __def__ method much more. -- Markus

1 0

shutil.trash()
by Virgil Dupras May 13, 2013

May 13, 2013

Hi all, A while ago, I've developed this library, send2trash ( https://bitbucket.org/hsoft/send2trash ), which can send files to trash on Mac OS X, Windows, and any platform that conforms to FreeDesktop. The current version uses ctypes, but earlier versions were straight C modules. I was wondering if you think this has a place in the stdlib, maybe as "shutil.trash()"? Virgil Dupras

4 3

improve Idle
by Alexandre Boulay May 13, 2013

May 13, 2013

I think that could be a good idea to put colored dots on idle's scroll bar for each def or class created, each got its own color, that's not a big conceptual improvement but that could be helpfull to show the structure, show what is what and which class is in which class

10 11

external editor commands in python REPL
by Yury Selivanov May 13, 2013

May 13, 2013

Hi all, While working on a repl-like shell for one of our applications, I implemented a buffer-editing mechanism similar to PostgreSQL shell. The idea is that you have special shell commands, starting with '\': - '\e' opens an external editor - '\r' clears the buffer - etc Playing with it for couple of days proved that it makes working with the REPL much more convenient. Here's a quick screencast: https://dl.dropboxusercontent.com/u/21052/repl.mov And if anyone wants to try it, here is the hack: https://gist.github.com/1st1/5569467 Run it under python 3: '$ python3 shell.py' I think it would be nice if we can include similar feature to the standard python REPL, as it makes it more convenient and OTOH starting python line with '\' is almost always illegal. - Yury

3 5

Re: [Python-ideas] Anonymous blocks (again):
by Juancarlo Añez May 13, 2013

May 13, 2013

Although I said I was over this never-ending topic, I thought I might say that: let: def block(): # do stuff in: closure(block) Would be well understood by many in in the Python and functional-language communities. Cheers, On Sun, May 12, 2013 at 7:43 PM, Martin Morrison < martin.morrrison(a)cantab.net> wrote: > On 13 May 2013, at 00:40, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > > Now, do you have any constructive feedback on the PEP that still accounts > for Python's lack of a standard location for passing callables to > functions, or is this reaction simply a matter of "I don't want to have to > type 'f' twice because I don't have to do that in other languages"? > > Moving past the outright negative feedback, and having only just seen the > PEP, the proposed syntax did strike me as awkward and unintuitive. Maybe > there is some explanation for why decorator-like syntax was used - if so, > please do link me so I can read up. > > What struck me though is that the proposed syntax limits the ability to > have multiple "anonymous blocks" within a single statement. Instead, I was > thinking some syntax like the following might be nicer: > > in x = do_something(in_arg, success_hdlr, error_hdlr): > def success_hdlr(result): > ... # Do something with result > def error_hdlr(error): > ... # Do something with error > > That is instead of a decorator-like syntax, make the "in" keyword reusable > to introduce a new block, whose "argument" is a statement that can forward > reference some names, which are then defined within the block. This allows > multiple temporary names to be defined (each in a separate statement within > the block). > > Some further thought is required on whether only def (and maybe class) > statements should be allowed within the "in" block. Although I guess > there's technically nothing wrong with: > > in x = y + z: > y = 12 > z = 30 > > Other than it's a very verbose way of doing something simple. ;-) But > maybe there are more useful examples? > > Cheers, > Martin > > Regards, > Nick. > > -- Juancarlo *Añez*

1 0

Add a .pop() method to ZipFile
by Daniel Holth May 11, 2013

May 11, 2013

Check out this efficient way to remove the last file from any ordinary zip file. class TruncatingZipFile(zipfile.ZipFile): """ZipFile that can pop files off the end. This works for ordinary zip files that do not contain non-ZIP data interleaved between the compressed files.""" def pop(self): """Truncate the last file off this zipfile.""" if not self.fp: raise RuntimeError( "Attempt to pop from ZIP archive that was already closed") last = self.infolist().pop() del self.NameToInfo[last.filename] self.fp.seek(last.header_offset, os.SEEK_SET) self.fp.truncate() self._didModify = True

9 17

Re: [Python-ideas] Implicit string literal concatenation considered harmful?
by Jonathan Eunice May 10, 2013

May 10, 2013

Implicit line concatenation is one of those rare places where Python turns an oddly blind eye to the "Explicit is better than implicit" rule it otherwise loves. I can't speak to how much inconvenience/breakage it would cause, but deprecating it would seem to increase the language's "coherence" with--or at least, adherence to--its principles. But I have no real stake in it; I seldom if ever use the construct. I prefer a trick I learned in Perl: Using a "here document" cleanup function. This allows multi-line literal strings to be stated in a program, at whatever level of indentation is appropriate for code clarity, and the indentation to be automatically removed. For example, using [textdata](https://pypi.python.org/pypi/textdata): from textdata import * data = lines(""" There was an old woman who lived in a shoe. She had so many children, she didn't know what to do; """) will result in: ['There was an old woman who lived in a shoe.', "She had so many children, she didn't know what to do;"] This is dedented, but also had some blank lines at the start and end removed (the blanks make Python formatting look good, but might gunk up subsequent processing). `lines` can also do what implicit concatenation does: data = lines(join=True, text=""" this that """) gives a purely concatenated result: 'thisthat' The `join` kwarg, given a string, will join on that string. Some edge-case options aside, being able to state indented literal strings without fussing with line-by-line quoting is the jewel. Especially if Python's implicit line concatenation is going to be deprecated, you might consider adding an "ease of multi-line string specification" function like `lines` to the standard library (in `textwrap`?) to ease the passing.

1 0

2 ideas for `concurrent.futures`
by Ram Rachum May 10, 2013

May 10, 2013

I have 2 ideas I thought about for the `futures` module: 1. A method `Executor.filter` that will be to the built-in `filter` what `Executor.filter` is to the built-in `map`. 2. A keyword argument `join_on_exit` to `Executor.__init__`, with a default of `False`. When `True`, upon `Executor.__exit__` all futures will be run to completion. This'll be useful to avoid the `Cannot schedule new futures after shutdown` exception without manually exhausting all the iterators returned by e.g. `Executor.map`. What do you think? Thanks, Ram.

1 0

improving C structs layout
by Charles-François Natali May 10, 2013

May 10, 2013

Hi, I was recently looking at the PyThreadState data structure (for issue #17912, but it's unimportant), and noticed that the layout of the members leaves some holes (due to alignment). While it doesn't import too much for PyThreadState (because of trailing padding), I wondered whether other structures in the code base could benefit from a better layout. So I ran pahole [1], and found the following structures: $ pahole -P python PyMemberDef 40 32 8 wrapperbase 56 48 8 unicode_formatter_t 136 128 8 _expr 56 52 4 _stmt 72 68 4 _excepthandler 40 36 4 _node 40 32 8 compiler_unit 448 440 8 tok_state 992 984 8 The first column is the current size, and the second column the size after a more judicious layout. For example. Before: $ pahole -C wrapperbase python struct wrapperbase { char * name; /* 0 8 */ int offset; /* 8 4 */ /* XXX 4 bytes hole, try to pack */ void * function; /* 16 8 */ wrapperfunc wrapper; /* 24 8 */ char * doc; /* 32 8 */ int flags; /* 40 4 */ /* XXX 4 bytes hole, try to pack */ PyObject * name_strobj; /* 48 8 */ /* size: 56, cachelines: 1, members: 7 */ /* sum members: 48, holes: 2, sum holes: 8 */ /* last cacheline: 56 bytes */ }; After: $ pahole -C wrapperbase -R python struct wrapperbase { char * name; /* 0 8 */ int offset; /* 8 4 */ int flags; /* 12 4 */ void * function; /* 16 8 */ wrapperfunc wrapper; /* 24 8 */ char * doc; /* 32 8 */ PyObject * name_strobj; /* 40 8 */ /* size: 48, cachelines: 1, members: 7 */ /* last cacheline: 48 bytes */ }; /* saved 8 bytes! */ While some of the structs above aren't worth the trouble (like tok_state), I think some might be interesting candidates. This could lead to reduced memory usage (well, of course it depends on the number of instances), and better cache usage/locality of reference. So what do you think, is it worth it? cf [1] https://github.com/acmel/dwarves

12 17