Separate compilation and friends
Hi. There is growing interest about PyPy and especially about extension modules. Apparently there are some people (like Alex) that are willing to write modules in RPython that should not go to the main tree. Since separate compilation is considered hard, how hard would it be to provide separate loading? This would mean you still compile the whole interpreter, but you can load the module from a compiled PyPy that didn't have this option on. Cheers, fijal
On a related note, how hard is it to "freeze" the translator/compiler state of a given pypy version just before it begins to read extension modules and distribute that, it would speed up module development a lot. It would be a quick equivalent of distributing headers for a C library. d. On 14 February 2011 18:13, Maciej Fijalkowski <fijall@gmail.com> wrote:
Hi.
There is growing interest about PyPy and especially about extension modules. Apparently there are some people (like Alex) that are willing to write modules in RPython that should not go to the main tree. Since separate compilation is considered hard, how hard would it be to provide separate loading? This would mean you still compile the whole interpreter, but you can load the module from a compiled PyPy that didn't have this option on.
Cheers, fijal _______________________________________________ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
On 15/02/11 03:41, Dima Tisnek wrote:
On a related note, how hard is it to "freeze" the translator/compiler state of a given pypy version just before it begins to read extension modules and distribute that, it would speed up module development a lot. It would be a quick equivalent of distributing headers for a C library.
this is hard, because the compilation of the modules is intermixed with the compilation of the rest of the interpreter for each phase: we have (roughly) something like: - annotation of the interpreter - annotation of the modules - rtyping of the interpreter - rtyping of the modules - etc. etc. ciao, Anto
On 15 February 2011 01:10, Antonio Cuni <anto.cuni@gmail.com> wrote:
On 15/02/11 03:41, Dima Tisnek wrote:
On a related note, how hard is it to "freeze" the translator/compiler state of a given pypy version just before it begins to read extension modules and distribute that, it would speed up module development a lot. It would be a quick equivalent of distributing headers for a C library.
this is hard, because the compilation of the modules is intermixed with the compilation of the rest of the interpreter for each phase: we have (roughly) something like:
- annotation of the interpreter - annotation of the modules - rtyping of the interpreter - rtyping of the modules - etc. etc.
ciao, Anto
Yeah I figured as much, I was wondering if it could be changed like this: - annotation of the interpreter, save state (1) - rtyping of the interpreter, shouldn't depend on modules here, save state (2) - etc. - annotation of the modules, using state from 1 - rtyping of the modules, using state from 1,2 - etc. I assume here that modules don't introduce dependencies into the iterpreter. I guess in the long run this ought to be the case, right? If this is possible, it would be a useful quick hack to separate module build from main build. If it's still very hard, then some else is in order. I'd love to play with this myself, but I don't have enough ram for a full build ;-( d.
On Tue, Feb 15, 2011 at 09:17, Dima Tisnek <dimaqq@gmail.com> wrote:
On 15 February 2011 01:10, Antonio Cuni <anto.cuni@gmail.com> wrote:
On 15/02/11 03:41, Dima Tisnek wrote:
On a related note, how hard is it to "freeze" the translator/compiler state of a given pypy version just before it begins to read extension modules and distribute that, it would speed up module development a lot. It would be a quick equivalent of distributing headers for a C library.
this is hard, because the compilation of the modules is intermixed with the compilation of the rest of the interpreter for each phase: we have (roughly) something like:
- annotation of the interpreter - annotation of the modules
How much time would be saved by saving the annotation results?
- rtyping of the interpreter - rtyping of the modules - etc. etc.
ciao, Anto
Yeah I figured as much, I was wondering if it could be changed like this:
- annotation of the interpreter, save state (1) - rtyping of the interpreter, shouldn't depend on modules here, save state (2) - etc. - annotation of the modules, using state from 1 - rtyping of the modules, using state from 1,2 - etc.
I assume here that modules don't introduce dependencies into the iterpreter. I guess in the long run this ought to be the case, right?
I don't think you can guarantee this. Type inference is global, and you might need a user for each API to better infer its type. Maybe uses of an API in testcases allow fully inferring their types, but I'd guess not. However, what is true in general is that if less specific types are inferred, that affects just performance, not correctness (I don't know if that's true of PyPy, but you ought to be able to pass "object"s around). Maybe the slowdown is insignificant, maybe it is a huge problem, maybe few annotations can save the day. However, it is still not clear (to me) where previous efforts stopped. Is it hard to: 1) devise an algorithm like Dima proposed or to 2) implement it (because of too much code to change and limited manpower) or to 3) or to have a small performance loss? Per-file separate compilation would likely fall into 3), because too little type inference would happen, isn't it?
If this is possible, it would be a useful quick hack to separate module build from main build. If it's still very hard, then some else is in order. I'd love to play with this myself, but I don't have enough ram for a full build ;-(
-- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/
On 15 February 2011 01:32, Paolo Giarrusso <pgiarrusso@mathematik.uni-marburg.de> wrote:
On Tue, Feb 15, 2011 at 09:17, Dima Tisnek <dimaqq@gmail.com> wrote:
On 15 February 2011 01:10, Antonio Cuni <anto.cuni@gmail.com> wrote:
On 15/02/11 03:41, Dima Tisnek wrote:
On a related note, how hard is it to "freeze" the translator/compiler state of a given pypy version just before it begins to read extension modules and distribute that, it would speed up module development a lot. It would be a quick equivalent of distributing headers for a C library.
this is hard, because the compilation of the modules is intermixed with the compilation of the rest of the interpreter for each phase: we have (roughly) something like:
- annotation of the interpreter - annotation of the modules
How much time would be saved by saving the annotation results?
- rtyping of the interpreter - rtyping of the modules - etc. etc.
ciao, Anto
Yeah I figured as much, I was wondering if it could be changed like this:
- annotation of the interpreter, save state (1) - rtyping of the interpreter, shouldn't depend on modules here, save state (2) - etc. - annotation of the modules, using state from 1 - rtyping of the modules, using state from 1,2 - etc.
I assume here that modules don't introduce dependencies into the iterpreter. I guess in the long run this ought to be the case, right?
I don't think you can guarantee this. Type inference is global, and you might need a user for each API to better infer its type. Maybe uses of an API in testcases allow fully inferring their types, but I'd guess not.
However, what is true in general is that if less specific types are inferred, that affects just performance, not correctness (I don't know if that's true of PyPy, but you ought to be able to pass "object"s around). Maybe the slowdown is insignificant, maybe it is a huge problem, maybe few annotations can save the day.
Correct me if I'm wong, but I assume that rpython and python type inferences are quite different. After all we don't have user code when we translate pypy itself, do we? Coming back to rpython-only discussion: I suppose one option woud be include type speciation in the modules themselves, or is that an overkill? Say if I write an ubermodule with uberdatum class in rpython and then use standard library bisect in that module, currenty as all modules and core are compiled together, an uberdatum-specific bisect loop can be made. If we move ubermodule out, it cannot be done. At the same time, if modules carried own versions of everything including all builtins, those modules would be just too big. Something in between would be a fixed "ABI" where core and stdlib are one blob and each 3rd party module is another blob, and everything that crosses the boundary is treated as a "pyobject".
However, it is still not clear (to me) where previous efforts stopped. Is it hard to: 1) devise an algorithm like Dima proposed or to 2) implement it (because of too much code to change and limited manpower) or to 3) or to have a small performance loss?
Per-file separate compilation would likely fall into 3), because too little type inference would happen, isn't it?
If this is possible, it would be a useful quick hack to separate module build from main build. If it's still very hard, then some else is in order. I'd love to play with this myself, but I don't have enough ram for a full build ;-(
-- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/
On Tue, Feb 15, 2011 at 09:10, Antonio Cuni <anto.cuni@gmail.com> wrote:
On 15/02/11 03:41, Dima Tisnek wrote:
On a related note, how hard is it to "freeze" the translator/compiler state of a given pypy version just before it begins to read extension modules and distribute that, it would speed up module development a lot. It would be a quick equivalent of distributing headers for a C library.
this is hard, because the compilation of the modules is intermixed with the compilation of the rest of the interpreter for each phase: we have (roughly) something like:
- annotation of the interpreter - annotation of the modules - rtyping of the interpreter - rtyping of the modules - etc. etc.
[Separate loading] This would mean you still compile the whole interpreter, but you can load the module from a compiled PyPy that didn't have this option on. At least as far as I understand, if "interpreter" means just the interpreter "core", as in Antonio's mail, this seems just as hard. Otherwise, if "interpreter" includes modules, this extension would not accomodate the needs of authors of external modules: one would have download all external modules and compile them together, but at runtime one could save the RAM footprint of external modules. Is that
Thanks for explaining the problem. IIRC, rtyping is a global analysis because for performance you perform type inference globally, so that for instance if a module allows inferring that a field is a number, that information is propagated to the original definition. Depending on what Maciej meant, his proposal might be as hard: the relevant problem? Cheers, -- Paolo Giarrusso - Ph.D. Student http://www.informatik.uni-marburg.de/~pgiarrusso/
On 15 February 2011 12:13, Maciej Fijalkowski <fijall@gmail.com> wrote:
Hi.
There is growing interest about PyPy and especially about extension modules. Apparently there are some people (like Alex) that are willing to write modules in RPython that should not go to the main tree. Since separate compilation is considered hard, how hard would it be to provide separate loading? This would mean you still compile the whole interpreter, but you can load the module from a compiled PyPy that didn't have this option on.
By "this option" you mean --withmod-xxx ? -- William Leslie
On 15 February 2011 12:13, Maciej Fijalkowski <fijall@gmail.com> wrote:
Hi.
There is growing interest about PyPy and especially about extension modules. Apparently there are some people (like Alex) that are willing to write modules in RPython that should not go to the main tree. Since separate compilation is considered hard, how hard would it be to provide separate loading? This would mean you still compile the whole interpreter, but you can load the module from a compiled PyPy that didn't have this option on.
0. What do you do about linking? Are rpython class and function names mangled consistently enough that a module compiled against one patch-level runtime with one set of options will link against another patch version with some different options? 1. Is it reasonable to ensure that *all* symbols that may be visible to an extension module are exported by the runtime? f may be inlined into g, but if f may still be callable by an extension module, it must be available by being exported static, too. 2. I always assumed this ("separate loading", ie, common translation) was the intended way to do separate compilation in rpython *anyway*; but there is one nagging thing. In order to define the boundary of the compilation, you already need to declare the interface of the module. Maybe not the annotations of the arguments, but at least which functions, when the annotator sees them, should be placed in your ".so". For the specific case of modules in the pypy python runtime this can be the interpleveldefs attribute of Module objects, but such functions all seem to have known signature annotation; they take a space and a number of wrapped arguments and return a wrapped object. Armed with this and the signature annotations of the functions in the core runtime, it is reasonable to expect that we can determine which functions belong in the ".so". Exactly how those core annotations are obtained doesn't need to be set in stone - obtained within the same translation or annotation performed earlier are both reasonable and not incompatible places to start. In both cases, the annotation is *computed* in the usual way. The alternative is defining the interface exported by the runtime explicitly, which is a mammoth task, and I don't think anyone has suggested it: is that what you are arguing against, fijal? There are still some details, such as how you export JitCodes from functions in your module, and what it means to do so - but nothing prohibitive that I can see. On 15 February 2011 19:32, Paolo Giarrusso <pgiarrusso@mathematik.uni-marburg.de> wrote:
On Tue, Feb 15, 2011 at 09:17, Dima Tisnek <dimaqq@gmail.com> wrote:
I assume here that modules don't introduce dependencies into the iterpreter. I guess in the long run this ought to be the case, right?
I don't think you can guarantee this. Type inference is global, and you might need a user for each API to better infer its type. Maybe uses of an API in testcases allow fully inferring their types, but I'd guess not.
If this does happen, it makes the callee part of a public interface, which should probably be explicitly annotated. I think checking for this case (where annotation in the extension would generalise the type signature of a dependency) is not too difficult.
However, what is true in general is that if less specific types are inferred, that affects just performance, not correctness (I don't know if that's true of PyPy, but you ought to be able to pass "object"s around). Maybe the slowdown is insignificant, maybe it is a huge problem, maybe few annotations can save the day.
Annotation widens, rather than narrows: overly specific types are more likely to be inferred.
However, it is still not clear (to me) where previous efforts stopped. Is it hard to: 1) devise an algorithm like Dima proposed or to 2) implement it (because of too much code to change and limited manpower) or to 3) or to have a small performance loss?
I understood that there was a lot of design to be done and there were other priorities. Devising an algorithm is not difficult, specifying it in terms of our existing annotation & flow model is slightly more so.
Per-file separate compilation would likely fall into 3), because too little type inference would happen, isn't it?
I suspect you are thinking of accidentally boxing interp-level integers or something, but that is an impossible condition (the dreaded SomeObject annotation). If not, where do you think a loss would come from? -- William Leslie
On 15 February 2011 01:13, Maciej Fijalkowski <fijall@gmail.com> wrote:
Hi.
There is growing interest about PyPy and especially about extension modules. Apparently there are some people (like Alex) that are willing to write modules in RPython that should not go to the main tree.
As a side note I think Alex's code *should* go in the main tree. It is really *needed* to work with django and pypy (at least for those using postgres which is a significant proportion of django users) and it would be much better [for pypy] if that "just worked" out of the box. Michael
Since separate compilation is considered hard, how hard would it be to provide separate loading? This would mean you still compile the whole interpreter, but you can load the module from a compiled PyPy that didn't have this option on.
Cheers, fijal _______________________________________________ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
-- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html
participants (6)
-
Antonio Cuni -
Dima Tisnek -
Maciej Fijalkowski -
Michael Foord -
Paolo Giarrusso -
William ML Leslie