Hello everybody, Here is a proposal for organizing our files and particularly putting them in relation. As the Python tests are "unit tests" I suggest the name "unit" for a body of work composed of a couple of related files. A unit is roughly something as big as a Python module, but could also be used for something like built-in types. For example: * 'Builtins module' unit: module/test/test_builtin.py CPython's original unit test file module/builtin.py module/builtin_app.py maybe CPython's __builtin__ module for the interface (docstrings...) * 'Integer object' unit: objspace/std/test/test_integer.py objspace/std/test/test_intobject.py objspace/std/intobject.py objspace/std/intobject_app.py A unit is something explicitely defined somewhere. It starts abstractedly, like with only the tests and the interface of the module (function signatures, docstrings), and it is later "completed", i.e. (possibly incrementially) implemented, with new implementation-specific tests and docs. Example assuming Python syntax, using subclassing for "completing": class BuiltinsModule(pypy.unit.ModuleUnit): "Expected interface for the Python __builtin__ module." modulename = "__builtin__" test1 = pypy.unit.Test("module/test/test_builtin.py") rawinterface = pypy.unit.InterfaceFromCPython("__builtin__") #...and possibly somewhere else... class MyBuiltins(path.to.the.above.BuiltinsModule): impl = pypy.unit.RPythonImplementation("./my_builtin.py") implapp = pypy.unit.PurePythonImplementation("./my_builtin_app.py") impltest = pypy.unit.Test("./test/test_my_builtin.py") maintainer = "arigo" # some tests are known to fail right now # (the following line is done with a __get__ method on pypy.unit.Test # so that we can refer to MyBuiltins' status of test1) MyBuiltins.test1.known_failing += ['test_zip', 'test_unichr'] # some functions I have not implemented right now MyBuiltins.rawinterface.known_missing += ['filter', 'reduce'] # Note that the base BuiltinsModule class has no implementation, # hence tests won't be tested directly on it. # Also note that there are only a couple of magic attribute names, # like 'modulename' and 'maintainer'; all the pypy.unit.Xxx() # instances can be found by enumerating all attributes. The purpose of the classes of pypy.unit is to define the "roles" that each file can assume within that unit. They contains the magic to collect the data together when we actually want to run the tests or pypy itself. For example, RPythonImplementation would search for all functions defined in the specified file and insert them as "built-in function objects" into the proper application-level module (or maybe not all the functions, but only the ones specially tagged, or only the ones that are said to be part of the module interface). Instead of the above Python syntax for the definition of units, it could also be done with a dedicated syntax; here XML: module/builtinsmodule.xml: <unit kind="Module"> <modulename>__builtin__</modulename> <test id="test1">module/test/test_builtin.py</test> <interface id="raw" source="CPython">__builtin__</interface> </unit> module/arigo/mybuiltins.xml: <unit extends="module/builtinsmodule.xml"> <doc>blah blah blah</doc> <implementation lang="RPython">./my_builtins.py</implementation> <implementation lang="PurePython">./my_builtins_app.py</implementation> <test>./test/test_my_builtins.py</test> <maintainer>arigo</maintainer> <status> <failing test_id="test1"> test_zip test_unichr </failing> <missing interface_id="raw"> filter reduce </missing> </status> </unit> Not too bad either, althought having the power of the Python language to express things like 'known_failing += [...]' is more flexible. Opinions? Opinions on the whole idea are welcome too, of course! As you might have noticed I have located mybuiltins.xml in "module/arigo/", following Laura's idea of separating our in-development files from the main ones. I have not specified here anything about the actual "release" of the files, i.e. the process of moving them from these per-user subdirectories into the mainstream "module/". For now I'd say it could be as easy as moving up the .xml file and the files it links to, and that we'll see later if we need to agree on a more precise policy for that. Also note that contributing a test file that you expect to work on any implementation of a module means just adding a line in the (base) unit corresponding to the empty module interface. We should then precisely define the pypy.unit role classes, e.g. how Test() files are called, what function signatures RPythonImplementation() expect (including how to do the PyArg_ParseTuple() trick more declaratively), and how interfaces and possibly doc files are written. We also probably need another kind of implementation, like LibraryImplementation, a file containing just ctypes-like declarations of library function calls (e.g. for the posix module). I hope that this could give the frame that everybody is waiting for to start the actual work! A bientôt, Armin.
Hello. I thought I'd offer this observation: There seems to be a lot of activity between a python source and producing the result of "executing" it. Even in CPython, there are implicitly checks on the environment and for the existence of .pyc files, etc., and conditional compilation based on these. Descriptions of pypy aspirations suggest more complex and conditional behavior in the process of "executing" a python script. As a thought experiment, how do you react to thinking of the total process as controlled by a hidden pypy_make and associated files (python source, default make-file template, etc.)? Even if pypy_make doesn't literally exist, I'm wondering if the make concept (being familiar) would help organize all the activities and dependencies, especially since various targeting options are desired, and various optimization strategies are to be coordinated, etc., etc., including possible use of alternate implementation languages for modules/extensions. I thought maybe the concept of python execution/interpreting via implicit pypy_make might make factoring/modularizing the various functions easier and more flexible than a more monolithic interpreter concept, but this is just a thought. Thus pypy script.py would perhaps act like pypy_make results -f pypy_run.mak -s script.py under the covers, and pypy_run.mak (and script.py.mak if present?) could accumulate the declarative glue to express all the relationships. Maybe scons could be cannibalized, if this were a viable implementation strategy? Anyway, just a thought, FWIW. Regards, Bengt (Back from 2 wks on a sunny beach ;-)
Hello Bengt, On Thu, Mar 13, 2003 at 07:53:10PM -0800, Bengt Richter wrote:
Yes, but I am not sure there is a need for a 'make'-based running process. A similar result can be acheived by correctly defining and linking together Python code containing declarative classes, which is what I am trying to do with the "unit" stuff. In other words, as we are using Python to control how parts are tied to each other, we already have much more flexibility than if we were using C (as in CPython).
I would prefer this glue to be written in a form that is easily accessible to Python algorithms. The SCons project could be useful here, too; I think I will more seriously look at it and see if it could be used to declare what I have in mind for "units", namely extensible bags of files, each playing a parametrizable role within the unit. A bientôt, Armin.
Hello. I thought I'd offer this observation: There seems to be a lot of activity between a python source and producing the result of "executing" it. Even in CPython, there are implicitly checks on the environment and for the existence of .pyc files, etc., and conditional compilation based on these. Descriptions of pypy aspirations suggest more complex and conditional behavior in the process of "executing" a python script. As a thought experiment, how do you react to thinking of the total process as controlled by a hidden pypy_make and associated files (python source, default make-file template, etc.)? Even if pypy_make doesn't literally exist, I'm wondering if the make concept (being familiar) would help organize all the activities and dependencies, especially since various targeting options are desired, and various optimization strategies are to be coordinated, etc., etc., including possible use of alternate implementation languages for modules/extensions. I thought maybe the concept of python execution/interpreting via implicit pypy_make might make factoring/modularizing the various functions easier and more flexible than a more monolithic interpreter concept, but this is just a thought. Thus pypy script.py would perhaps act like pypy_make results -f pypy_run.mak -s script.py under the covers, and pypy_run.mak (and script.py.mak if present?) could accumulate the declarative glue to express all the relationships. Maybe scons could be cannibalized, if this were a viable implementation strategy? Anyway, just a thought, FWIW. Regards, Bengt (Back from 2 wks on a sunny beach ;-)
Hello Bengt, On Thu, Mar 13, 2003 at 07:53:10PM -0800, Bengt Richter wrote:
Yes, but I am not sure there is a need for a 'make'-based running process. A similar result can be acheived by correctly defining and linking together Python code containing declarative classes, which is what I am trying to do with the "unit" stuff. In other words, as we are using Python to control how parts are tied to each other, we already have much more flexibility than if we were using C (as in CPython).
I would prefer this glue to be written in a form that is easily accessible to Python algorithms. The SCons project could be useful here, too; I think I will more seriously look at it and see if it could be used to declare what I have in mind for "units", namely extensible bags of files, each playing a parametrizable role within the unit. A bientôt, Armin.
participants (2)
-
Armin Rigo -
Bengt Richter