[Python-Dev] Release of astoptimizer 0.3

[Victor Stinner]

2012/9/12 Serhiy Storchaka <storchaka at gmail.com>:
>>> set([x for ...]) => {x for ...}
>>> dict([(k, v) for ...]) => {k: v for ...}
>>> dict((k, v) for ...) => {k: v for ...}
>>> ''.join([s for ...]) => ''.join(s for ...)
>>> a.extend([s for ...]) => a.extend(s for ...)
>>
>> These optimizations look correct.
>
> Actually generator can be slower list comprehension. Especially on Python2.
> I think this is an opportunity to optimize the work with generators.

I checked with timeit, and yes: generators are slower :-/ I will
revert this "optimization".

>>> (f(x) for x in a) => map(f, a)
>>> (x.y for x in a) => map(operator.attrgetter('y'), a)
>>> (x[0] for x in a) => map(operator.itemgetter(0), a)
>>> (2 * x for x in a) => map((2).__mul__, a)
>>> (x in b for x in a) => map(b.__contains__, a)
>>> map(lambda x: x.strip(), a) => (x.strip() for x in a)
>>
>> Is it faster? :-)

Benchmark using iterable=tuple(range(n)) and f=str (see attached script).

Python version: 2.7.3 (default, Aug 1 2012, 05:16:07) [GCC 4.6.3]
CPU model: Intel(R) Core(TM) i5 CPU 661 @ 3.33GHz
Platform: Linux-3.2.0-30-generic-pae-i686-with-Ubuntu-12.04-precise
Bits: int=32, long=32, long long=64, pointer=32

[ 3 items ]
679 ns: list comprehesion
1.08 us (+59%): itertools.imap
1.42 us (+109%): generator

[ 10 items ]
1.6 us: itertools.imap
1.64 us: list comprehesion
2.26 us (+41%): generator

[ 1000 items ]
112 us: itertools.imap
144 us (+29%): list comprehesion
156 us (+40%): generator

[ 1000000 items ]
142 ms: itertools.imap
183 ms (+29%): generator
186 ms (+31%): list comprehesion

---

Python version: 3.2.3 (default, May 3 2012, 15:54:42) [GCC 4.6.3]
CPU model: Intel(R) Core(TM) i5 CPU 661 @ 3.33GHz
Platform: Linux-3.2.0-30-generic-pae-i686-with-Ubuntu-12.04-precise
Bits: int=32, long=32, long long=64, pointer=32

[ 3 items ]
1.04 us: list comprehesion
1.21 us (+17%): map
1.51 us (+45%): generator


[ 10 items ]
2.02 us: map
2.29 us (+13%): list comprehesion
2.68 us (+33%): generator


[ 1000 items ]
132 us: map
166 us (+25%): list comprehesion
183 us (+38%): generator


[ 1000000 items ]
182 ms: map
229 ms (+26%): generator
251 ms (+38%): list comprehesion

--

> Yes, significantly for large sequences. But this transformation is not safe
> in general case. For short sequences possible regression (cost of "map" name
> lookup and function call).

So except for very small dataset, map (itertools.imap) is always
faster. List comprehension cannot be replaced with map() because map()
doesn't set the iterator variable to the last item if the iterable.

>>> x in ['i', 'em', 'cite'] => x in {'i', 'em', 'cite'}
> Agree, it applicable if x is proven str. At least list can be replaced by
> tuple.
>>> (...)
>>> x == 'i' or x == 'em' or x == 'cite'] => x in {'i', 'em', 'cite'}
>>> (...)
>>> x = x + 1 => x += 1
>>> x = x + ' ' => x += ' '

Well, type inference would permit more optimizations. It is not implemented yet.

>>> for ...: f.write(...) => __fwrite = f.write; for ...: __fwrite(...)
>>
>> f.write lookup cannot be optimized.
>
> Yes, it is a dangerous transformation and it is difficult to prove its
> safety. But name lookup is one of the main brakes of Python.

Oh sorry, I didn't read correctly the example. Yeah, such optimization
is common and it would help to have an option to enable it. Using type
inference, it may be possible to optimize safetly some cases (ex: if
you know that f is a file).

> Sorry, it should be 'x=%s' % repr(x) => 'x=%r' % (x,)

Ah ok, why not.

>>> range(0, x) => range(x)
>>
>> Is it faster?
>
> Slightly.

timeit gives me exactly the same timing.

> I personally would prefer a 2to3-like "modernizer" (as a separate utility
> and as plugins for the IDEs), which would have found some templates and
> offered replacing by a more modern, readable (and possibly effective)
> variant. The decision on the applicability of the transformation in the
> particular case remains for the human. For the automatic optimizer remain
> only simple transformations which deteriorate readability, and optimizations
> which cannot be expressed in the source code.

If the optimizer sees an interesting optimization but cannot decide if
one option is better than another, it may write an advice for the
developer. The developer can review the advice and decide which option
is the best.

Some patterns are faster or slower depending on the Python versions
:-/ The optimizer may be able to decide which pattern is the best for
the running Python version.

Victor
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bench_gen.py
Type: application/octet-stream
Size: 875 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120912/74724c76/attachment.obj>