Hi, are there any python construct that the jit will be able to compile into c-type array accesses? Consider the following test: l=0.0 for i in xrange(640,640*480): l+=img[i] intimg[i]=intimg[i-640]+l With the 1.3 release of the jit it executes about 20 times slower than a similar construction in C if I create the arrays using: import _rawffi RAWARRAY = _rawffi.Array('d') img=RAWARRAY(640*480, autofree=True) intimg=RAWARRAY(640*480, autofree=True) Using a list is about 40 times slower and using array.array is about 400 times slower. Any suggestion on how to improve the performance of these kind of constructions? Thanx. -- Håkan Ardö
Hey. There is a variety of reasons why those behave like this (array module is in PyPy written in Python for example, using _rawffi). There is a branch that plans to fix that for all lists, but that's not finished yet. On Thu, Jul 1, 2010 at 8:02 AM, Hakan Ardo <hakan@debian.org> wrote:
Hi, are there any python construct that the jit will be able to compile into c-type array accesses? Consider the following test:
l=0.0 for i in xrange(640,640*480): l+=img[i] intimg[i]=intimg[i-640]+l
With the 1.3 release of the jit it executes about 20 times slower than a similar construction in C if I create the arrays using:
import _rawffi RAWARRAY = _rawffi.Array('d') img=RAWARRAY(640*480, autofree=True) intimg=RAWARRAY(640*480, autofree=True)
Using a list is about 40 times slower and using array.array is about 400 times slower. Any suggestion on how to improve the performance of these kind of constructions?
Thanx.
-- Håkan Ardö _______________________________________________ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
Hi, On Thu, Jul 01, 2010 at 04:02:30PM +0200, Hakan Ardo wrote:
are there any python construct that the jit will be able to compile into c-type array accesses? Consider the following test:
l=0.0 for i in xrange(640,640*480): l+=img[i] intimg[i]=intimg[i-640]+l
This is still implemented as a list of Python objects (as expected, because the JIT cannot prove that we won't suddenly try to put something else than a float in the same list). Using _rawffi.Array('d') directly is the best option right now. I'm not sure why the array.array module is 400 times slower, but it's definitely slower given that it's implemented at app-level using a _rawffi.Array('c') and doing the conversion by itself (for some partially stupid reasons like doing the right kind of error checking). A bientot, Armin.
On Thu, Jul 1, 2010 at 9:28 AM, Armin Rigo <arigo@tunes.org> wrote:
Hi,
On Thu, Jul 01, 2010 at 04:02:30PM +0200, Hakan Ardo wrote:
are there any python construct that the jit will be able to compile into c-type array accesses? Consider the following test:
l=0.0 for i in xrange(640,640*480): l+=img[i] intimg[i]=intimg[i-640]+l
This is still implemented as a list of Python objects (as expected, because the JIT cannot prove that we won't suddenly try to put something else than a float in the same list).
Using _rawffi.Array('d') directly is the best option right now. I'm not sure why the array.array module is 400 times slower, but it's definitely slower given that it's implemented at app-level using a _rawffi.Array('c') and doing the conversion by itself (for some partially stupid reasons like doing the right kind of error checking).
A bientot,
Armin.
The main reason why _rawffi.Array is slow is that JIT does not look into that module, so there is wrapping and unwrapping going on. Relatively easy to fix I suppose, but _rawffi.Array was not meant to be used like that (array.array looks like a better candidate).
On Thu, Jul 1, 2010 at 10:35 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
On Thu, Jul 1, 2010 at 9:28 AM, Armin Rigo <arigo@tunes.org> wrote:
Hi,
On Thu, Jul 01, 2010 at 04:02:30PM +0200, Hakan Ardo wrote:
are there any python construct that the jit will be able to compile into c-type array accesses? Consider the following test:
l=0.0 for i in xrange(640,640*480): l+=img[i] intimg[i]=intimg[i-640]+l
This is still implemented as a list of Python objects (as expected, because the JIT cannot prove that we won't suddenly try to put something else than a float in the same list).
Using _rawffi.Array('d') directly is the best option right now. I'm not sure why the array.array module is 400 times slower, but it's definitely slower given that it's implemented at app-level using a _rawffi.Array('c') and doing the conversion by itself (for some partially stupid reasons like doing the right kind of error checking).
A bientot,
Armin.
The main reason why _rawffi.Array is slow is that JIT does not look into that module, so there is wrapping and unwrapping going on. Relatively easy to fix I suppose, but _rawffi.Array was not meant to be used like that (array.array looks like a better candidate). _______________________________________________ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
If array.array performance is important to your work, the array.py module looks like a good target for writing at interp level, and it's not too much code. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Voltaire "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me
OK, so making an interpreter level implementation of array.array seams like a good idea. Would it be possible to get the jit to remove the wrapping/unwrapping in that case to get better performance than _rawffi.Array('d'), which is already an interpreter level implementation? Are there some docs to get me started at writing interpreter level objects? I've had a look at _rawffi/array.py and am a bit confused about the W_Array.typedef = TypeDef('Array',...) construction. Maybe there is a easier example to start with? On Thu, Jul 1, 2010 at 5:40 PM, Alex Gaynor <alex.gaynor@gmail.com> wrote:
On Thu, Jul 1, 2010 at 10:35 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
On Thu, Jul 1, 2010 at 9:28 AM, Armin Rigo <arigo@tunes.org> wrote:
Hi,
On Thu, Jul 01, 2010 at 04:02:30PM +0200, Hakan Ardo wrote:
are there any python construct that the jit will be able to compile into c-type array accesses? Consider the following test:
l=0.0 for i in xrange(640,640*480): l+=img[i] intimg[i]=intimg[i-640]+l
This is still implemented as a list of Python objects (as expected, because the JIT cannot prove that we won't suddenly try to put something else than a float in the same list).
Using _rawffi.Array('d') directly is the best option right now. I'm not sure why the array.array module is 400 times slower, but it's definitely slower given that it's implemented at app-level using a _rawffi.Array('c') and doing the conversion by itself (for some partially stupid reasons like doing the right kind of error checking).
A bientot,
Armin.
The main reason why _rawffi.Array is slow is that JIT does not look into that module, so there is wrapping and unwrapping going on. Relatively easy to fix I suppose, but _rawffi.Array was not meant to be used like that (array.array looks like a better candidate). _______________________________________________ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
If array.array performance is important to your work, the array.py module looks like a good target for writing at interp level, and it's not too much code.
Alex
-- "I disapprove of what you say, but I will defend to the death your right to say it." -- Voltaire "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me _______________________________________________ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
-- Håkan Ardö
On Fri, Jul 2, 2010 at 12:24 AM, Hakan Ardo <hakan@debian.org> wrote:
OK, so making an interpreter level implementation of array.array seams like a good idea. Would it be possible to get the jit to remove the wrapping/unwrapping in that case to get better performance than _rawffi.Array('d'), which is already an interpreter level implementation?
Are there some docs to get me started at writing interpreter level objects? I've had a look at _rawffi/array.py and am a bit confused about the W_Array.typedef = TypeDef('Array',...) construction. Maybe there is a easier example to start with?
On Thu, Jul 1, 2010 at 5:40 PM, Alex Gaynor <alex.gaynor@gmail.com> wrote:
On Thu, Jul 1, 2010 at 10:35 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
On Thu, Jul 1, 2010 at 9:28 AM, Armin Rigo <arigo@tunes.org> wrote:
Hi,
On Thu, Jul 01, 2010 at 04:02:30PM +0200, Hakan Ardo wrote:
are there any python construct that the jit will be able to compile into c-type array accesses? Consider the following test:
l=0.0 for i in xrange(640,640*480): l+=img[i] intimg[i]=intimg[i-640]+l
This is still implemented as a list of Python objects (as expected, because the JIT cannot prove that we won't suddenly try to put something else than a float in the same list).
Using _rawffi.Array('d') directly is the best option right now. I'm not sure why the array.array module is 400 times slower, but it's definitely slower given that it's implemented at app-level using a _rawffi.Array('c') and doing the conversion by itself (for some partially stupid reasons like doing the right kind of error checking).
A bientot,
Armin.
The main reason why _rawffi.Array is slow is that JIT does not look into that module, so there is wrapping and unwrapping going on. Relatively easy to fix I suppose, but _rawffi.Array was not meant to be used like that (array.array looks like a better candidate). _______________________________________________ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
If array.array performance is important to your work, the array.py module looks like a good target for writing at interp level, and it's not too much code.
Alex
-- "I disapprove of what you say, but I will defend to the death your right to say it." -- Voltaire "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me _______________________________________________ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
-- Håkan Ardö _______________________________________________ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
I'd take a look at the cStringIO module, it's a decent example of the APIs (and not too much code). FWIW one thing to note is that array uses the struct module, which is also pure python. I believe it's possible to still use that with an interp-level module, but it may just become another bottle neck, just something to consider. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Voltaire "The people's good is the highest law." -- Cicero "Code can always be simpler than you think, but never as simple as you want" -- Me
Hi Alex, On Fri, Jul 02, 2010 at 12:40:21AM -0500, Alex Gaynor wrote:
FWIW one thing to note is that array uses the struct module, which is also pure python.
No: we have a pure Python version, but in a normally compiled pypy-c, there is an interp-level version of 'struct' too. A bientot, Armin.
Hi Fijal, On Thu, Jul 01, 2010 at 09:35:17AM -0600, Maciej Fijalkowski wrote:
The main reason why _rawffi.Array is slow is that JIT does not look into that module, so there is wrapping and unwrapping going on. Relatively easy to fix I suppose, but _rawffi.Array was not meant to be used like that (array.array looks like a better candidate).
If you mean "better candidate" for being fast right now, then you missed my point: our array.array module is implemented on top of _rawffi.Array. If you mean "better candidate" for being optimizable given some work, then yes, I agree that the array module is a good target. A bientot, Armin.
On Fri, Jul 2, 2010 at 2:17 AM, Armin Rigo <arigo@tunes.org> wrote:
Hi Fijal,
On Thu, Jul 01, 2010 at 09:35:17AM -0600, Maciej Fijalkowski wrote:
The main reason why _rawffi.Array is slow is that JIT does not look into that module, so there is wrapping and unwrapping going on. Relatively easy to fix I suppose, but _rawffi.Array was not meant to be used like that (array.array looks like a better candidate).
If you mean "better candidate" for being fast right now, then you missed my point: our array.array module is implemented on top of _rawffi.Array. If you mean "better candidate" for being optimizable given some work, then yes, I agree that the array module is a good target.
By "better candidate" I mean that having JIT see _rawffi might mean some struggle for it to understand what's going on with raw pointers and writing array in interp-level would be better.
Hi Fijal, On Fri, Jul 02, 2010 at 02:19:02AM -0600, Maciej Fijalkowski wrote:
By "better candidate" I mean that having JIT see _rawffi might mean some struggle for it to understand what's going on with raw pointers and writing array in interp-level would be better.
Ah, right. Armin.
participants (4)
-
Alex Gaynor -
Armin Rigo -
Hakan Ardo -
Maciej Fijalkowski