Hi Everyone, Just synced to head this morning and noticed a couple of errors in the c translation process. This first is that on windows, a couple of headers are not available. I fixed this by patching locally with: Index: translator/c/extfunc_include.h =================================================================== --- translator/c/extfunc_include.h (revision 14483) +++ translator/c/extfunc_include.h (working copy) @@ -6,9 +6,9 @@ /******************************************************************/ -#include <unistd.h> +/*#include <unistd.h> #include <sys/types.h> -#include <sys/stat.h> +#include <sys/stat.h>*/ #include <fcntl.h> #include <time.h> #ifndef PATH_MAX Then, when the tests are run I get: $ ./test_all.py translator/c/ ============================= test process starts ============================= testing-mode: inprocess executable: c:\Python24\python.exe (2.4.1-final-0) using py lib: c:\Documents and Settings\YoungB\dist\py <rev unknown> translator\c\test\test_annotated.py[19] ................... translator\c\test\test_backendoptimized.py[39] ....................................... translator\c\test\test_database.py[17] ................. translator\c\test\test_exception.py[3] ... translator\c\test\test_extfunc.py[4] F... translator\c\test\test_genc.py[8] ........ translator\c\test\test_lltyped.py[2] .. translator\c\test\test_notype.py[33] ................................. translator\c\test\test_operation.py[1] . translator\c\test\test_support.py[7] ....... translator\c\test\test_symboltable.py[1] . translator\c\test\test_typed.py[38] ...................................... _______________________________________________________________________________ _________________________ entrypoint: test_time_clock _________________________ def test_time_clock(): def does_stuff(): return time.clock() f1 = compile(does_stuff, []) t0 = time.clock() t1 = f1() assert type(t1) is float t2 = time.clock() E assert t0 <= t1 <= t2
assert 165.0 <= 2.5683609740123038e-005
[c:\Documents and Settings\YoungB\dist\pypy\translator\c\test\test_extfunc.py:16] - - - - - - - - - - - test_time_clock: recorded stderr - - - - - - - - - - - - [annrpython] (pypy.translator.c.test.test_extfunc:9) does_stuff -> SomeFloat() [annrpython] (pypy.rpython.rclass:575) ll_runtime_type_info__objectPtr -> SomePtr(ll_ptrtype=<* RuntimeTypeInfo (opaque)>) [annrpython] (pypy.rpython.module.ll_time:14) ll_time_clock -> SomeFloat() [annrpython] (pypy.rpython.rclass:568) ll_issubclass__object_vtablePtr_object_vtablePtr -> SomeBool(const=True) [annrpython] (pypy.rpython.rclass:568) ll_issubclass__object_vtablePtr_object_vtablePtr -> SomeBool() [annrpython] (pypy.rpython.rclass:565) ll_type__objectPtr -> SomePtr(ll_ptrtype=<* Struct object_vtable { parenttypeptr: * Struct objec t_vtable { ... }, rtti: * RuntimeTypeInfo (opaque), name: * Array of Char }>) [annrpython] (pypy.rpython.exceptiondata:120) ll_pyexcclass2exc__PyObjectPtr -> SomePtr(const=<* struct object { typeptr=... }>, ll_ptr type=<* GcStruct object { typeptr: * Struct object_vtable { parenttypeptr: * Struct object_vtable { ... }, rtti: * RuntimeTypeInfo (opa que), name: * Array of Char } }>) [annrpython] (pypy.rpython.exceptiondata:120) ll_pyexcclass2exc__PyObjectPtr -> SomePtr(ll_ptrtype=<* GcStruct object { typeptr: * Stru ct object_vtable { parenttypeptr: * Struct object_vtable { ... }, rtti: * RuntimeTypeInfo (opaque), name: * Array of Char } }>) [annrpython] (pypy.rpython.exceptiondata:53) ll_raise_OSError__Signed -> SomeImpossibleValue() _______________________________________________________________________________ =========== tests finished: 171 passed, 1 failed in 263.86 seconds ============ I think this is because when clock is called in the extension it thinks it is the first time and returns the very small value. I don't really know though. I hope you all had fun at the sprint! Cheers, Ben
Hi Ben, On Mon, Jul 11, 2005 at 10:10:28AM +0100, Ben.Young@risk.sungard.com wrote:
This first is that on windows, a couple of headers are not available. I fixed this by patching locally with:
Thanks. I commented out the #includes for now, as they come anyway with <Python.h>. Later, we will need to re-enable them to become independent of CPython, with lots of #ifdefs and possibly a 'configure'-based build system similar to CPython's.
_______________________________________________________________________________ _________________________ entrypoint: test_time_clock
This failure should be fixed now (by copying all of CPython's time.clock() implementation).
I hope you all had fun at the sprint!
Definitely :-) Thanks, and thanks too for keeping a close eye on PyPy :-) A bientot, Armin
Thanks for that! Unfortunately thats exposes a new problem, as there is a function called RaiseException included by windows.h! Maybe all your macros should be prefixed by PyPy_ or something. It sounds like you are getting close to the translate_pypy target. Would it be possible for you to give a summary of the problems you appear to be having with the PBC code (as evidenced by some of the commit messages)? I'm just interested in what the problem appears to be, and whether it's just something you're going to have to live with, or if there is an architectural solution? I don't really understand the PBC side of things. Is there a reason why the object hierarchy can't just mirror the way CPython does it? Or would that be too slow? Thanks for all your help! Cheers, Ben Armin Rigo <arigo@tunes.org> wrote on 12/07/2005 17:40:40:
Hi Ben,
On Mon, Jul 11, 2005 at 10:10:28AM +0100, Ben.Young@risk.sungard.com wrote:
This first is that on windows, a couple of headers are not available. I fixed this by patching locally with:
Thanks. I commented out the #includes for now, as they come anyway with <Python.h>. Later, we will need to re-enable them to become independent of CPython, with lots of #ifdefs and possibly a 'configure'-based build system similar to CPython's.
_______________________________________________________________________________
_________________________ entrypoint: test_time_clock
This failure should be fixed now (by copying all of CPython's time.clock() implementation).
I hope you all had fun at the sprint!
Definitely :-) Thanks, and thanks too for keeping a close eye on PyPy :-)
A bientot,
Armin
Hi Ben, On Wed, Jul 13, 2005 at 10:03:28AM +0100, Ben.Young@risk.sungard.com wrote:
Unfortunately thats exposes a new problem, as there is a function called RaiseException included by windows.h! Maybe all your macros should be prefixed by PyPy_ or something.
Argh. I see. Yes, using a common prefix more systematically looks like a good idea.
It sounds like you are getting close to the translate_pypy target. Would it be possible for you to give a summary of the problems you appear to be having with the PBC code (as evidenced by some of the commit messages)? I'm just interested in what the problem appears to be, and whether it's just something you're going to have to live with, or if there is an architectural solution?
The PBCs are messy in general, I don't think we will find a nice a clean solution. At least currently, the annotator says that everything is a "PBC" if it is any kind of user-defined prebuilt object, but this is only part of the problem. The real messiness comes from the fact that the PBCs are somehow the RPython objects that have the least obvious C equivalents: the Python classes, some prebuilt instances, and functions (with some of their full Python parameter passing semantics). Moreover they come out of the annotator in families (a SomePBC is a set of PBCs), so we have to handle such families in various ways depending on how the objects in the family are actually used. The flow graph viewer doesn't show all this, because it doesn't really display complex low-level types and constants nicely. Not sure if and how to improve that...
I don't really understand the PBC side of things. Is there a reason why the object hierarchy can't just mirror the way CPython does it? Or would that be too slow?
Yes, we are indeed trying for each kind of PBC family to figure out the best C-ish equivalent. A family of functions should become just a C function pointer; a family of classes should become a vtable pointer; etc. We always planned to do that, and our RPython code is written accordingly, but the devil's in the details... It looks like we are mostly done, though. According to http://codespeak.net/svn/pypy/dist/pypy/translator/goal/ISSUES.txt the last remaining problem is about conversions from small to larger PBC families, which should be easy (typically, no conversion is needed e.g. to cast a pointer-to-function to a pointer-to-function-that-can-point-to-more-functions-than-the-previous-one). A bientot, Armin
Armin Rigo wrote:
Hi Ben,
On Wed, Jul 13, 2005 at 10:03:28AM +0100, Ben.Young@risk.sungard.com wrote:
Unfortunately thats exposes a new problem, as there is a function called RaiseException included by windows.h! Maybe all your macros should be prefixed by PyPy_ or something.
Argh. I see. Yes, using a common prefix more systematically looks like a good idea.
It sounds like you are getting close to the translate_pypy target. Would it be possible for you to give a summary of the problems you appear to be having with the PBC code (as evidenced by some of the commit messages)? I'm just interested in what the problem appears to be, and whether it's just something you're going to have to live with, or if there is an architectural solution?
The PBCs are messy in general, I don't think we will find a nice a clean solution. At least currently, the annotator says that everything is a "PBC" if it is any kind of user-defined prebuilt object, but this is only part of the problem. The real messiness comes from the fact that the PBCs are somehow the RPython objects that have the least obvious C equivalents: the Python classes, some prebuilt instances, and functions (with some of their full Python parameter passing semantics). Moreover they come out of the annotator in families (a SomePBC is a set of PBCs), so we have to handle such families in various ways depending on how the objects in the family are actually used.
The flow graph viewer doesn't show all this, because it doesn't really display complex low-level types and constants nicely. Not sure if and how to improve that...
I don't really understand the PBC side of things. Is there a reason why the object hierarchy can't just mirror the way CPython does it? Or would that be too slow?
Yes, we are indeed trying for each kind of PBC family to figure out the best C-ish equivalent. A family of functions should become just a C function pointer; a family of classes should become a vtable pointer; etc. We always planned to do that, and our RPython code is written accordingly, but the devil's in the details...
It looks like we are mostly done, though. According to http://codespeak.net/svn/pypy/dist/pypy/translator/goal/ISSUES.txt the last remaining problem is about conversions from small to larger PBC families, which should be easy (typically, no conversion is needed e.g. to cast a pointer-to-function to a pointer-to-function-that-can-point-to-more-functions-than-the-previous-one).
at least the PyPy codebase doesn't show such kind of conversion issues anymore. Now we are with some functionality missing the rtyper and nasty issues related to faking code and appearing PyObjects. I should updated the ISSUES file, with the list of error that translate_pypy with -t-insist produces.
Armin Rigo <arigo@tunes.org> wrote on 21/07/2005 18:51:28:
Hi Ben,
Snip ...
The flow graph viewer doesn't show all this, because it doesn't really display complex low-level types and constants nicely. Not sure if and how to improve that...
I don't really understand the PBC side of things. Is there a reason
why
the object hierarchy can't just mirror the way CPython does it? Or would that be too slow?
Yes, we are indeed trying for each kind of PBC family to figure out the best C-ish equivalent. A family of functions should become just a C function pointer; a family of classes should become a vtable pointer; etc. We always planned to do that, and our RPython code is written accordingly, but the devil's in the details...
So is a family of classes the set of any classes that can be assigned to a particular instance variable, even if they dont share a common base? e.g: class A: pass class B: pass foo = somebool ? A() : B() Would it be simpler to require that all instances must at least be annotated to a common base, and fail to annotate if a not-base-defined method is called on the instance. Or is that really what you are already doing!? Do function PBCs all have to have the same signature, or can it vary? Anyway, thanks for the reply! As illuminating as ever. I guess with the rtyping, the first flush of exciting coding has gone and you are down to the grinding out bugs stage. Are you pleased with the way things have gone or are there any chages you would make to the architecture now? Cheers, Ben P.S os.ftruncate doesn't exist on windows I'm afraid, so most of the tests fail at the moment
A bientot,
Armin
Hi Ben, On Fri, Jul 22, 2005 at 09:18:56AM +0100, Ben.Young@risk.sungard.com wrote:
So is a family of classes the set of any classes that can be assigned to a particular instance variable, even if they dont share a common base?
e.g:
class A: pass
class B: pass
foo = somebool ? A() : B()
Would it be simpler to require that all instances must at least be annotated to a common base, and fail to annotate if a not-base-defined method is called on the instance. Or is that really what you are already doing!?
Indeed, we track each class that can be in a particular variable, instead of (at this point) falling back to a common class, requiring that it exists. While this would not be absolutely necessary for classes, we need the notion of "set of values" anyway for functions and for prebuilt constant instances.
Do function PBCs all have to have the same signature, or can it vary?
It can vary to some extent. We have code in rpython/normalizecalls.py that analyzes calls from a single place that can go to a family of functions, and that "normalizes" the functions -- i.e. modify their signature and insert conversions at the beginning of the functions if needed, so that they all get exactly the same signature.
Anyway, thanks for the reply! As illuminating as ever. I guess with the rtyping, the first flush of exciting coding has gone and you are down to the grinding out bugs stage. Are you pleased with the way things have gone or are there any chages you would make to the architecture now?
Given that we succeeded in translating the whole of PyPy yesterday -- sorry for the long delay again in answering -- I guess that yes, I'm pretty much pleased in general with the way everything has gone :-)
P.S os.ftruncate doesn't exist on windows I'm afraid, so most of the tests fail at the moment
Thanks for the report. It still fails on Windows, indeed... A bientot, Armin
Thats great news. Congratuations to all of you! Cheers, Ben Armin Rigo <arigo@tunes.org> wrote on 31/07/2005 20:24:12:
Hi Ben,
On Fri, Jul 22, 2005 at 09:18:56AM +0100, Ben.Young@risk.sungard.com wrote:
So is a family of classes the set of any classes that can be assigned to a particular instance variable, even if they dont share a common base?
e.g:
class A: pass
class B: pass
foo = somebool ? A() : B()
Would it be simpler to require that all instances must at least be annotated to a common base, and fail to annotate if a not-base-defined
method is called on the instance. Or is that really what you are already doing!?
Indeed, we track each class that can be in a particular variable, instead of (at this point) falling back to a common class, requiring that it exists. While this would not be absolutely necessary for classes, we need the notion of "set of values" anyway for functions and for prebuilt constant instances.
Do function PBCs all have to have the same signature, or can it vary?
It can vary to some extent. We have code in rpython/normalizecalls.py that analyzes calls from a single place that can go to a family of functions, and that "normalizes" the functions -- i.e. modify their signature and insert conversions at the beginning of the functions if needed, so that they all get exactly the same signature.
Anyway, thanks for the reply! As illuminating as ever. I guess with the rtyping, the first flush of exciting coding has gone and you are down to the grinding out bugs stage. Are you pleased with the way things have gone or are there any chages you would make to the architecture now?
Given that we succeeded in translating the whole of PyPy yesterday -- sorry for the long delay again in answering -- I guess that yes, I'm pretty much pleased in general with the way everything has gone :-)
P.S os.ftruncate doesn't exist on windows I'm afraid, so most of the tests fail at the moment
Thanks for the report. It still fails on Windows, indeed...
A bientot,
Armin
participants (3)
-
Armin Rigo -
Ben.Young＠risk.sungard.com -
Samuele Pedroni