Mailman 3 July 2010 - pypy-dev

Interpreter level array implementation
by Hakan Ardo 04 Jul '10

04 Jul '10

Hi, we got the simplest possible interpreter level implementation of an array-like object running (in the interplevel-array branch) and it executes my previous example about 2 times slower than optimized C. Attached is the trace generated by the following example: img=array(640*480); l=0; i=0; while i<640*480: l+=img[i] i+=1 a simplified version of that trace is: 1. [p0, p1, p2, p3, i4, p5, p6, p7, p8, p9, p10, f11, i] 2. i14 = int_lt(i, 307200) 3. guard_true(i14, descr=<Guard1>) 4. guard_nonnull_class(p10, 145745952, descr=<Guard2>) 5. img = getfield_gc(p10, descr=<GcPtrFieldDescr 8>) 6. f17 = getarrayitem_gc(img, i, descr=<FloatArrayDescr>) 7. f18 = float_add(f11, f17) 8. i20 = int_add_ovf(i, 1) 9. guard_no_overflow(, descr=<Guard3>) # 10. i23 = getfield_raw(149604768, descr=<SignedFieldDescr 0>) 11. i25 = int_add(i23, 1) 12. setfield_raw(149604768, i25, descr=<SignedFieldDescr 0>) 13. i28 = int_and(i25, -2131755008) 14. i29 = int_is_true(i28) 15. guard_false(i29, descr=<Guard4>) 16. jump(p0, p1, p2, p3, 27, ConstPtr(ptr31), ConstPtr(ptr32), ConstPtr(ptr33), p8, p9, p10, f18, i20) Does these operation more or less correspond to assembler instructions? I guess that the extra overhead here as compared to the the C version would be line 4, 5, 9 and 10-15. What's 10-15 all about? I guess that most of these additional operation would not affect the performance of more complicated loops as they will only occur once per loop (although combining the guard on line 9 with line 3 might be a possible optimization)? Line 4 will appear once for each array used in the loop and line 5 once for every array access, right? Can the array implementation be designed in someway that would not generate line 5 above? Or would it be possible to get rid of it by some optimization? -- Håkan Ardö

8 13

Re: [pypy-dev] array performace?
by Paolo Giarrusso 04 Jul '10

04 Jul '10

On Fri, Jul 2, 2010 at 10:55, <Ben.Young(a)sungard.com> wrote: >> On Fri, Jul 2, 2010 at 2:26 AM, <Ben.Young(a)sungard.com> wrote: >> >> On Fri, Jul 2, 2010 at 1:47 AM, Paolo Giarrusso >> > <p.giarrusso(a)gmail.com> >> >> wrote: >> >> > On Fri, Jul 2, 2010 at 08:04, Maciej Fijalkowski <fijall(a)gmail.com> >> > wrote: >> >> >> On Thu, Jul 1, 2010 at 1:18 PM, Hakan Ardo <Hakan(a)ardoe.net> wrote: >> >> >>> OK, so making an interpreter level implementation of array.array >> > seams >> >> >>> like a good idea. Would it be possible to get the jit to remove >> > the >> >> >>> wrapping/unwrapping in that case to get better performance than >> >> >>> _rawffi.Array('d'), which is already an interpreter level >> >> >>> implementation? >> >> >> >> >> >> it should work mostly out of the box (you can also try this for >> >> >> _rawffi.array part of module, if you want to). It's probably enough >> > to >> >> >> enable module in pypy/module/pypyjit/policy.py so JIT can have a >> > look >> >> >> there. In case of _rawffi, probably a couple of hints for the jit >> > to >> >> >> not look inside some functions (which do external calls for >> > example) >> >> >> should also be needed, since for example JIT as of now does not >> >> >> support raw mallocs (using C malloc and not our GC). >> >> > >> >> >> Still, making an >> >> >> array module interp-level is probably the sanest approach. >> >> > >> >> > That might be a bad sign. >> >> > For CPython, people recommend to write extensions in C for >> >> > performance, i.e. to make them less maintainable and understandable >> >> > for performance. >> >> > A good JIT should make this unnecessary in as many cases as >> > possible. >> >> > Of course, the array module might be an exception, if it's a single >> >> > case. >> >> > But performance 20x slower than C, with a JIT, is a big warning, >> > since >> >> > fast interpreters are documented to be (in general) just 10x slower >> >> > than C. >> >> >> >> There is a lot of unsupported claims in your sentences, however, >> >> that's not my point. >> >> >> > >> > That's a little harsh. When the JIT was originally developed it was >> > envisaged that it would be faster to re-write code to app level to give >> > speed-ups. If that's changed that's fine, but it's not an "unsupported >> > claim" >> > >> > Ben >> > >> >> Unsupported claim is for example that fast interpreters are 10x slower >> than C. That's the only unsupported claim, but it comes from "The Structure and Performance of Eﬃcient Interpreters". I studied that as a student on VM, you are writing one, so I (unconsciously) guessed that everybody knows that paper - I know that's a completely broken way of writing, but I didn't spot it. >>On what exactly? Did he write this particular benchmark in C >> and in fast interpreter to compare? Another unsupported claim is that >> JIT is 20x slower than C here. I did not claim that - I am aware that it is not even JITted. I complain against the lack of JITting. >> Array module is not even JITted, >> because it's based on _rawffi which itself operates on low-level >> pointers which JIT does not want to deal with. I would say that instead of doing manual annotations or rewriting at the interp-level (which doesn't scale), it would be overall simpler to make the JIT learn itself how to deal with those calls (i.e. inline everything around, leave the external call as a call), once and for all. What you suggest below might be a way to do it. >> That's exactly the >> reason why JIT doesn't look into _rawffi module and making it look >> there doesn't sound like a good idea (instead, we're trying to replace >> it with something JIT-friendly that knows how to do FFI calls into C, >> there is a summer of code project). Well, at the abstraction level I'm speaking, it sounds like there in the end, the JIT will be able to do what is needed. I am not aware of the details. But then, at the end of that project, it seems to me that it should be possible to write the array module in pure Python using this new FFI interface and have the JIT look at it, shouldn't it? I do not concentrate on array specifically - rewriting a few modules at interpreter level is fine. But as a Python developer I should have no need for that. >> All I'm trying to say is that there are valid reasons that array >> module should be on interpreter level and none of this has anything to >> do with incapabilities of the JIT. > Fair enough, and I do see your point, but I think Paolo comment was not aimed at array, just the implication (in this case) that to get performance you need to re-write in rpython. I think his point in general is correct, even if he picked the wrong example to mention it :) (and his 20x claim comes from the original email, so I don't think it's entirely unsupported) Thanks for understanding my point. I'm unsure whether an ideal JIT could allow leaving array at the app-level (and I noted also in the original mail that I was unsure on this). > Of course in this case I'm sure there are good reasons, but it is certainly interesting to see the push towards more rpython code than app-level. I guess that's because the JIT can "see" and accelerate rpython code too I believe, so it’s win-win (because of the code size issues and things like that) > Incidentally, is there a reason that geninterped code is so bloated compared to rpython code that looks like it could have been generated from the app-level equivalent? Would there be a way of annotating the app-level code so that when it's geninterped it's as tight as the equivalent rpython? -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/

7 11

PyPy Speed
by Ben.Young＠sungard.com 02 Jul '10

02 Jul '10

http://speed.pypy.org/overview/ seems to have been unavailable for the last couple of days. It gives a 500 whenever I visit it Ben Young - Senior Software Engineer SunGard - Enterprise House, Vision Park, Histon, Cambridge, CB24 9ZR Tel +44 1223 266042 - Main +44 1223 266100 - http://www.sungard.com/ CONFIDENTIALITY: This email (including any attachments) may contain confidential, proprietary and privileged information, and unauthorized disclosure or use is prohibited. If you received this email in error, please notify the sender and delete this email from your system. Thank you.

3 6

Re: [pypy-dev] array performace?
by Maciej Fijalkowski 02 Jul '10

02 Jul '10

On Fri, Jul 2, 2010 at 2:26 AM, <Ben.Young(a)sungard.com> wrote: >> On Fri, Jul 2, 2010 at 1:47 AM, Paolo Giarrusso > <p.giarrusso(a)gmail.com> >> wrote: >> > On Fri, Jul 2, 2010 at 08:04, Maciej Fijalkowski <fijall(a)gmail.com> > wrote: >> >> On Thu, Jul 1, 2010 at 1:18 PM, Hakan Ardo <Hakan(a)ardoe.net> wrote: >> >>> OK, so making an interpreter level implementation of array.array > seams >> >>> like a good idea. Would it be possible to get the jit to remove > the >> >>> wrapping/unwrapping in that case to get better performance than >> >>> _rawffi.Array('d'), which is already an interpreter level >> >>> implementation? >> >> >> >> it should work mostly out of the box (you can also try this for >> >> _rawffi.array part of module, if you want to). It's probably enough > to >> >> enable module in pypy/module/pypyjit/policy.py so JIT can have a > look >> >> there. In case of _rawffi, probably a couple of hints for the jit > to >> >> not look inside some functions (which do external calls for > example) >> >> should also be needed, since for example JIT as of now does not >> >> support raw mallocs (using C malloc and not our GC). >> > >> >> Still, making an >> >> array module interp-level is probably the sanest approach. >> > >> > That might be a bad sign. >> > For CPython, people recommend to write extensions in C for >> > performance, i.e. to make them less maintainable and understandable >> > for performance. >> > A good JIT should make this unnecessary in as many cases as > possible. >> > Of course, the array module might be an exception, if it's a single >> > case. >> > But performance 20x slower than C, with a JIT, is a big warning, > since >> > fast interpreters are documented to be (in general) just 10x slower >> > than C. >> >> There is a lot of unsupported claims in your sentences, however, >> that's not my point. >> > > That's a little harsh. When the JIT was originally developed it was > envisaged that it would be faster to re-write code to app level to give > speed-ups. If that's changed that's fine, but it's not an "unsupported > claim" > > Ben > Unsupported claim is for example that fast interpreters are 10x slower than C. On what exactly? Did he write this particular benchmark in C and in fast interpreter to compare? Another unsupported claim is that JIT is 20x slower than C here. Array module is not even JITted, because it's based on _rawffi which itself operates on low-level pointers which JIT does not want to deal with. That's exactly the reason why JIT doesn't look into _rawffi module and making it look there doesn't sound like a good idea (instead, we're trying to replace it with something JIT-friendly that knows how to do FFI calls into C, there is a summer of code project). All I'm trying to say is that there are valid reasons that array module should be on interpreter level and none of this has anything to do with incapabilities of the JIT. Cheers, fijal

1 0

array performace?
by Hakan Ardo 02 Jul '10

02 Jul '10

Hi, are there any python construct that the jit will be able to compile into c-type array accesses? Consider the following test: l=0.0 for i in xrange(640,640*480): l+=img[i] intimg[i]=intimg[i-640]+l With the 1.3 release of the jit it executes about 20 times slower than a similar construction in C if I create the arrays using: import _rawffi RAWARRAY = _rawffi.Array('d') img=RAWARRAY(640*480, autofree=True) intimg=RAWARRAY(640*480, autofree=True) Using a list is about 40 times slower and using array.array is about 400 times slower. Any suggestion on how to improve the performance of these kind of constructions? Thanx. -- Håkan Ardö

4 10

Re: [pypy-dev] array performace?
by Maciej Fijalkowski 02 Jul '10

02 Jul '10

On Thu, Jul 1, 2010 at 1:18 PM, Hakan Ardo <Hakan(a)ardoe.net> wrote: > OK, so making an interpreter level implementation of array.array seams > like a good idea. Would it be possible to get the jit to remove the > wrapping/unwrapping in that case to get better performance than > _rawffi.Array('d'), which is already an interpreter level > implementation? it should work mostly out of the box (you can also try this for _rawffi.array part of module, if you want to). It's probably enough to enable module in pypy/module/pypyjit/policy.py so JIT can have a look there. In case of _rawffi, probably a couple of hints for the jit to not look inside some functions (which do external calls for example) should also be needed, since for example JIT as of now does not support raw mallocs (using C malloc and not our GC). Still, making an array module interp-level is probably the sanest approach. > Are there some docs to get me started at writing interpreter level > objects? I've had a look at _rawffi/array.py and am a bit confused > about the W_Array.typedef = TypeDef('Array',...) construction. Maybe > there is a easier example to start with? TypeDef is a way to expose interpreter level (RPython) object to app-level (Python). It tells what methods there are what properties and what attributes. > > On Thu, Jul 1, 2010 at 5:40 PM, Alex Gaynor <alex.gaynor(a)gmail.com> wrote: >> On Thu, Jul 1, 2010 at 10:35 AM, Maciej Fijalkowski <fijall(a)gmail.com> wrote: >>> On Thu, Jul 1, 2010 at 9:28 AM, Armin Rigo <arigo(a)tunes.org> wrote: >>>> Hi, >>>> >>>> On Thu, Jul 01, 2010 at 04:02:30PM +0200, Hakan Ardo wrote: >>>>> are there any python construct that the jit will be able to compile >>>>> into c-type array accesses? Consider the following test: >>>>> >>>>> l=0.0 >>>>> for i in xrange(640,640*480): >>>>> l+=img[i] >>>>> intimg[i]=intimg[i-640]+l >>>> >>>> This is still implemented as a list of Python objects (as expected, >>>> because the JIT cannot prove that we won't suddenly try to put something >>>> else than a float in the same list). >>>> >>>> Using _rawffi.Array('d') directly is the best option right now. I'm not >>>> sure why the array.array module is 400 times slower, but it's definitely >>>> slower given that it's implemented at app-level using a _rawffi.Array('c') >>>> and doing the conversion by itself (for some partially stupid reasons like >>>> doing the right kind of error checking). >>>> >>>> >>>> A bientot, >>>> >>>> Armin. >>> >>> The main reason why _rawffi.Array is slow is that JIT does not look >>> into that module, so there is wrapping and unwrapping going on. >>> Relatively easy to fix I suppose, but _rawffi.Array was not meant to >>> be used like that (array.array looks like a better candidate). >>> _______________________________________________ >>> pypy-dev(a)codespeak.net >>> http://codespeak.net/mailman/listinfo/pypy-dev >> >> If array.array performance is important to your work, the array.py >> module looks like a good target for writing at interp level, and it's >> not too much code. >> >> Alex >> >> -- >> "I disapprove of what you say, but I will defend to the death your >> right to say it." -- Voltaire >> "The people's good is the highest law." -- Cicero >> "Code can always be simpler than you think, but never as simple as you >> want" -- Me >> _______________________________________________ >> pypy-dev(a)codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev > > > > -- > Håkan Ardö >

3 5

Re: [pypy-dev] New speed.pypy.org version
by Paolo Giarrusso 02 Jul '10

02 Jul '10

Hi Miquel, I'm quite busy (because of a paper deadline next Tuesday), sorry for not answering earlier. I was just struck by an idea: there is a stacked bar plot where the total bar is related to the geometric mean, such that it is normalization-invariant. But this graph _is_ complicated. It is a stacked plot of _logarithms_ of performance ratios? This way, the complete stacked bar shows the logarithm of the product, rather than their sum, i.e. the log of the (geometric mean)^N rather than their arithmetic mean. log of the (geometric mean)^N = N*log of the (geometric mean). Some simple maths (I didn't write it out, so please recheck!) seems to show that showing (a+b*log (ratio)), instead of log(ratio), gives still a fair comparison, obtaining N*a+b*N*log(geomean) = \Theta(log(geomean)). You need to put a and b because showing if the ratio is 1, log(1) is zero (b is the representation scale which is always there). About your workaround: I would like a table with the geometric mean of the ratios, where we get the real global performance ratio among the interpreters. As far as the results of your solution do not contradict that _real_ table, it should be a reasonable workaround (but I would embed the check in the code - otherwise other projects _will be_ bitten by that). Probably, I would like the website to offer such a table to users, and I would like a graph of the overall performance ratio over time (actually revisions). Finally, the docs of your web application should at the very least reference the paper and this conversation (if there's a public archive of the ML, as I think), and ideally explain the issue. Sorry for being too dense, maybe - if I was unclear, please tell me and I'll answer next week. Best regards, Paolo On Mon, Jun 28, 2010 at 11:21, Miquel Torres <tobami(a)googlemail.com> wrote: > Hi Paolo, > > I read the paper, very interesting. It is perfectly clear that to > calculate a normalized total only the geometric mean makes sense. > > However, a stacked bars plot shows the individual benchmarks so it > implicitly is an arithmetic mean. The only solution (apart from > removing the stacked charts and only offering total bars) is the > weighted approach. > > External weights are not very practical though. Codespeed is used by > other projects so an extra option would need to be added to the > settings to allow the introducing of arbitrary weights to benchmarks. > A bit cumbersome. I have an idea that may work. Take the weights from > a defined baseline so that the run times are equal, which is the same > as normalizing to a baseline. It would be the same as now, only that > you can't choose the normalization, it will be weighted (normalized) > according the default baseline (which you already can already > configure in the settings). > > You may say that it is still an arithmetic mean, but there won't be > conflicting results because there is only a single normalization. For > PyPy that would be cpython, and everything would make sense. > I know it is a work around, not a solution. If you think it is a bad > idea, the only other possibility is not to have stacked bars (as in > "showing individual benchmarks"). But I find them useful. Yes you can > see the individual benchmark results better in the normal bars chart, > but there you don't see visually which benchmarks take the biggest > part of the pie, which helps visualize what parts of your program need > most improving. > > What do you think? > > Regards, > Miquel > > > 2010/6/25 Paolo Giarrusso <p.giarrusso(a)gmail.com>: >> On Fri, Jun 25, 2010 at 19:08, Miquel Torres <tobami(a)googlemail.com> wrote: >>> Hi Paolo, >>> >>> I am aware of the problem with calculating benchmark means, but let me >>> explain my point of view. >>> >>> You are correct in that it would be preferable to have absolute times. Well, >>> you actually can, but see what it happens: >>> http://speed.pypy.org/comparison/?hor=true&bas=none&chart=stacked+bars >> >> Ahah! I didn't notice that I could skip normalization! This does not >> fully invalidate my point, however. >> >>> Absolute values would only work if we had carefully chosen benchmaks >>> runtimes to be very similar (for our cpython baseline). As it is, html5lib, >>> spitfire and spitfire_cstringio completely dominate the cummulative time. >> >> I acknowledge that (btw, it should be cumulative time, with one 'm', >> both here and in the website). >> >>> And not because the interpreter is faster or slower but because the >>> benchmark was arbitrarily designed to run that long. Any improvement in the >>> long running benchmarks will carry much more weight than in the short >>> running. >> >>> What is more useful is to have comparable slices of time so that the >>> improvements can be seen relatively over time. >> >> If you want to sum up times (but at this point, I see no reason for >> it), you should rather have externally derived weights, as suggested >> by the paper (in Rule 3). >> As soon as you take weights from the data, lots of maths that you need >> is not going to work any more - that's generally true in many cases in >> statistics. >> And the only way making sense to have external weights is to gather >> them from real world programs. Since that's not going to happen >> easily, just stick with the geometric mean. Or set an arbitrarily low >> weight, manually, without any math, so that the long-running >> benchmarks stop dominating the res. It's no fraud, since the current >> graph is less valid anyway. >> >>> Normalizing does that i >>> think. >> Not really. >> >>> It just says: we have 21 tasks which take 1 second to run each on >>> interpreter X (cpython in the default case). Then we see how other >>> executables compare to that. What would the geometric mean achieve here, >>> exactly, for the end user? >> >> You actually need the geomean to do that. Don't forget that the >> geomean is still a mean: it's a mean performance ratio which averages >> individual performance ratios. >> If PyPy's geomean is 0.5, it means that PyPy is going to run that task >> in 11.5 seconds instead of 21. To me, this sounds exactly like what >> you want to achieve. Moreover, it actually works, unlike what you use. >> >> For instance, ignore PyPy-JIT, and look only CPython and pypy-c (no >> JIT). Then, change the normalization among the two: >> http://speed.pypy.org/comparison/?exe=2%2B35,3%2BL&ben=1,2,3,4,5,6,7,8,9,10… >> http://speed.pypy.org/comparison/?exe=2%2B35,3%2BL&ben=1,2,3,4,5,6,7,8,9,10… >> with the current data, you get that in one case cpython is faster, in >> the other pypy-c is faster. >> It can't happen with the geomean. This is the point of the paper. >> >> I could even construct a normalization baseline $base such that >> CPython seems faster than PyPy-JIT. Such a base should be very fast >> on, say, ai (where CPython is slower), so that $cpython.ai/$base.ai >> becomes 100 and $pypyjit.ai/$base.ai becomes 200, and be very slow on >> other benchmarks (so that they disappear in the sum). >> >> So, the only difference I see is that geomean works, arithm. mean >> doesn't. That's why Real Benchmarkers use geomean. >> >> Moreover, you are making a mistake quite common among non-physicists. >> What you say makes sense under the implicit assumption that dividing >> two times gives something you can use as a time. When you say "Pypy's >> runtime for a 1 second task", you actually want to talk about a >> performance ratio, not about the time. In the same way as when you say >> "this bird runs 3 meters long in one second", a physicist would sum >> that up as "3 m/s" rather than "3 m". >> >>> I am not really calculating any mean. You can see that I carefully avoided >>> to display any kind of total bar which would indeed incur in the problem you >>> mention. That a stacked chart implicitly displays a total is something you >>> can not avoid, and for that kind of chart I still think normalized results >>> is visually the best option. >> >> But on a stacked bars graph, I'm not going to look at individual bars >> at all, just at the total: it's actually less convenient than in >> "normal bars" to look at the result of a particular benchmark. >> >> I hope I can find guidelines against stacked plots, I have a PhD >> colleague reading on how to make graphs. >> >> Best regards >> -- >> Paolo Giarrusso - Ph.D. Student >> http://www.informatik.uni-marburg.de/~pgiarrusso/ >> > -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/

2 2

Re: [pypy-dev] [pypy-svn] r75683 - in pypy/trunk: include lib-python/modified-2.5.2/distutils lib-python/modified-2.5.2/distutils/command pypy/_interfaces pypy/module/cpyext pypy/module/cpyext/test
by Maciej Fijalkowski 02 Jul '10

02 Jul '10

Hey. Any reason why we should copy .h files during translation and can't just have them there? Cheers, fijal On Wed, Jun 30, 2010 at 8:51 AM, <antocuni(a)codespeak.net> wrote: > Author: antocuni > Date: Wed Jun 30 16:51:13 2010 > New Revision: 75683 > > Added: > pypy/trunk/include/ (props changed) > pypy/trunk/include/README > Removed: > pypy/trunk/pypy/_interfaces/ > Modified: > pypy/trunk/lib-python/modified-2.5.2/distutils/command/build_ext.py > pypy/trunk/lib-python/modified-2.5.2/distutils/sysconfig_pypy.py > pypy/trunk/pypy/module/cpyext/api.py > pypy/trunk/pypy/module/cpyext/test/test_api.py > Log: > create a directory trunk/include to contains all the headers file. They are > automatically copied there from cpyext/include during translation. The > generated pypy_decl.h and pypy_macros.h are also put there, instead of the > now-gone pypy/_interfaces. > > The goal is to have the svn checkout as similar as possible as release > tarballs and virtualenvs, which have an include/ dir at the top > > > > Added: pypy/trunk/include/README > ============================================================================== > --- (empty file) > +++ pypy/trunk/include/README Wed Jun 30 16:51:13 2010 > @@ -0,0 +1,7 @@ > +This directory contains all the include files needed to build cpython > +extensions with PyPy. Note that these are just copies of the original headers > +that are in pypy/module/cpyext/include: they are automatically copied from > +there during translation. > + > +Moreover, pypy_decl.h and pypy_macros.h are automatically generated, also > +during translation. > > Modified: pypy/trunk/lib-python/modified-2.5.2/distutils/command/build_ext.py > ============================================================================== > --- pypy/trunk/lib-python/modified-2.5.2/distutils/command/build_ext.py (original) > +++ pypy/trunk/lib-python/modified-2.5.2/distutils/command/build_ext.py Wed Jun 30 16:51:13 2010 > @@ -167,7 +167,7 @@ > # for Release and Debug builds. > # also Python's library directory must be appended to library_dirs > if os.name == 'nt': > - self.library_dirs.append(os.path.join(sys.prefix, 'pypy', '_interfaces')) > + self.library_dirs.append(os.path.join(sys.prefix, 'include')) > if self.debug: > self.build_temp = os.path.join(self.build_temp, "Debug") > else: > > Modified: pypy/trunk/lib-python/modified-2.5.2/distutils/sysconfig_pypy.py > ============================================================================== > --- pypy/trunk/lib-python/modified-2.5.2/distutils/sysconfig_pypy.py (original) > +++ pypy/trunk/lib-python/modified-2.5.2/distutils/sysconfig_pypy.py Wed Jun 30 16:51:13 2010 > @@ -13,12 +13,7 @@ > > def get_python_inc(plat_specific=0, prefix=None): > from os.path import join as j > - cand = j(sys.prefix, 'include') > - if os.path.exists(cand): > - return cand > - if plat_specific: > - return j(sys.prefix, "pypy", "_interfaces") > - return j(sys.prefix, 'pypy', 'module', 'cpyext', 'include') > + return j(sys.prefix, 'include') > > def get_python_version(): > """Return a string containing the major and minor Python version, > > Modified: pypy/trunk/pypy/module/cpyext/api.py > ============================================================================== > --- pypy/trunk/pypy/module/cpyext/api.py (original) > +++ pypy/trunk/pypy/module/cpyext/api.py Wed Jun 30 16:51:13 2010 > @@ -45,11 +45,9 @@ > pypydir = py.path.local(autopath.pypydir) > include_dir = pypydir / 'module' / 'cpyext' / 'include' > source_dir = pypydir / 'module' / 'cpyext' / 'src' > -interfaces_dir = pypydir / "_interfaces" > include_dirs = [ > include_dir, > udir, > - interfaces_dir, > ] > > class CConfig: > @@ -100,9 +98,16 @@ > udir.join('pypy_macros.h').write("/* Will be filled later */") > globals().update(rffi_platform.configure(CConfig_constants)) > > -def copy_header_files(): > +def copy_header_files(dstdir): > + assert dstdir.check(dir=True) > + headers = include_dir.listdir('*.h') + include_dir.listdir('*.inl') > for name in ("pypy_decl.h", "pypy_macros.h"): > - udir.join(name).copy(interfaces_dir / name) > + headers.append(udir.join(name)) > + for header in headers: > + header.copy(dstdir) > + target = dstdir.join(header.basename) > + target.chmod(0444) # make the file read-only, to make sure that nobody > + # edits it by mistake > > _NOT_SPECIFIED = object() > CANNOT_FAIL = object() > @@ -881,7 +886,8 @@ > deco(func.get_wrapper(space)) > > setup_init_functions(eci) > - copy_header_files() > + trunk_include = pypydir.dirpath() / 'include' > + copy_header_files(trunk_include) > > initfunctype = lltype.Ptr(lltype.FuncType([], lltype.Void)) > @unwrap_spec(ObjSpace, str, str) > > Modified: pypy/trunk/pypy/module/cpyext/test/test_api.py > ============================================================================== > --- pypy/trunk/pypy/module/cpyext/test/test_api.py (original) > +++ pypy/trunk/pypy/module/cpyext/test/test_api.py Wed Jun 30 16:51:13 2010 > @@ -1,3 +1,4 @@ > +import py > from pypy.conftest import gettestobjspace > from pypy.rpython.lltypesystem import rffi, lltype > from pypy.interpreter.baseobjspace import W_Root > @@ -68,3 +69,13 @@ > api.PyPy_GetWrapped(space.w_None) > api.PyPy_GetReference(space.w_None) > > + > +def test_copy_header_files(tmpdir): > + api.copy_header_files(tmpdir) > + def check(name): > + f = tmpdir.join(name) > + assert f.check(file=True) > + py.test.raises(py.error.EACCES, "f.open('w')") # check that it's not writable > + check('Python.h') > + check('modsupport.inl') > + check('pypy_decl.h') > _______________________________________________ > pypy-svn mailing list > pypy-svn(a)codespeak.net > http://codespeak.net/mailman/listinfo/pypy-svn >

2 3

Improving Stackless/Coroutines implementation
by Gabriel Lavoie 02 Jul '10

02 Jul '10

Hello everyone, as a few knows here, I'm working heavily with PyPy's "stackless" module for my Master degree project to make it more distributed. Since I started to work full time on this project I've encountered a few bugs (mostly related to pickling of tasklets) and missing implementation details in the module. The latest problem I've encountered is to be able to detect when tasklet.kill() is called, within the tasklet being killed. With Stackless CPython, TaskletExit is raised and can be caught but this part wasn't really implemented in PyPy's stackless module. Since the module is implemented on top of coroutines and since coroutine.kill() is called within tasklet.kill(), the exception thrown by the coroutine implementation needs to be caught. Here's the problem: http://codespeak.net/pypy/dist/pypy/doc/stackless.html#coroutines - coro.kill() Kill coro by sending an exception to it. (At the moment, the exception is not visible to app-level, which means that you cannot catch it, and that try: finally: clauses are not honored. This will be fixed in the future.) The exception is not thrown at app level and a coroutine dies silently. Took a look at the code and I've been able to expose a CoroutineExit exception to app level on which I intend implementing TaskletExit correctly. I'm also able to catch the exception as expected but the code is not yet complete. Right now, I have a question on how to expose correctly the CoroutineExit and TaskletExit exceptions to app level. Here's what I did: W_CoroutineExit = _new_exception('CoroutineExit', W_Exception, 'Exit requested...') class AppCoroutine(Coroutine): # XXX, StacklessFlags): def __init__(self, space, state=None): # Some other code here # Exporting new exception to __builtins__ and "exceptions" modules self.w_CoroutineExit = space.gettypefor(W_CoroutineExit) space.setitem( space.exceptions_module.w_dict, space.new_interned_str('CoroutineExit'), self.w_CoroutineExit) space.setitem(space.builtin.w_dict, space.new_interned_str('CoroutineExit'), self.w_CoroutineExit) I talked about this on #pypy (IRC) but people weren't sure about exporting new names to __builtins__. On my side I wanted to make it look as most as possible as how Stackless CPython did it with TaskletExit, which is directly available in __builtins__. This would make code compatible with both Stackless Python and PyPy's stackless module. Also, exporting names this way would only make them appear in __builtins__ when the "_stackless" module is enabled (pypy-c built with --stackless). What are your opinions about it? (Maciej, I already know about yours! ;) Thank you very much, Gabriel (WildChild) -- Gabriel Lavoie glavoie(a)gmail.com

1 1

PyPy 1.3 released
by Maciej Fijalkowski 01 Jul '10

01 Jul '10

======================= PyPy 1.3: Stabilization ======================= Hello. We're please to announce release of PyPy 1.3. This release has two major improvements. First of all, we stabilized the JIT compiler since 1.2 release, answered user issues, fixed bugs, and generally improved speed. We're also pleased to announce alpha support for loading CPython extension modules written in C. While the main purpose of this release is increased stability, this feature is in alpha stage and it is not yet suited for production environments. Highlights of this release ========================== * We introduced support for CPython extension modules written in C. As of now, this support is in alpha, and it's very unlikely unaltered C extensions will work out of the box, due to missing functions or refcounting details. The support is disable by default, so you have to do:: import cpyext before trying to import any .so file. Also, libraries are source-compatible and not binary-compatible. That means you need to recompile binaries, using for example:: python setup.py build Details may vary, depending on your build system. Make sure you include the above line at the beginning of setup.py or put it in your PYTHONSTARTUP. This is alpha feature. It'll likely segfault. You have been warned! * JIT bugfixes. A lot of bugs reported for the JIT have been fixed, and its stability greatly improved since 1.2 release. * Various small improvements have been added to the JIT code, as well as a great speedup of compiling time. Cheers, Maciej Fijalkowski, Armin Rigo, Alex Gaynor, Amaury Forgeot d'Arc and the PyPy team

3 3