Welcome to the Minimal Python list, It appears (to me) that Armin and Christian are both busy for the weekend so i start some bootstrapping. First five paragraphs of background. The idea of actually starting to do "Minimal Python" arose from a thread in python-de (2003q1 first post). There soon was the idea of doing a sprint with the goal of releasing first bits. Armin, Christian and me realized early that opening up the discussion (instead of in private mails) is a very good idea as there is lots of experience and code out there. Nevertheless, i'd like the attendants of the planned sprint to decide which route they want to follow coding-wise eventually. We will propose some dates by the end of the month, btw. I think that - if in doubt - we follow the lead from CPython e.g. regarding coding style, language definition etc. Every deviation should be justified by the goals. I don't think we should follow a PEP process soon, though. During January i want to setup accounts and a cvs-repository. The involved policies will be liberal. ok. enough organizational things for now. Bootstrapping will be a hot topic for Minimal Python itself. We probably first want to use the CPython development environemnt to get going. We probably want to base it on the python-2.3 cvs tree. There is the idea of not using make/configure/automake but a simple understandable debuggable (read: python based) build environment. IOW words the number of dependencies for building Minimal Python should also be minimal. For avoiding the need of C-coded system modules there is interest to code a generalization of the "struct" module which allows *calling* functions at the C-level. This general C-"Extension" will be system-dependent. The gateway code between the "machine" and python probably needs to be coded in assembler: You can't "construct" a C-function call within ANSI-C. Both Microsoft and Apple have proprietary solutions for Java. (these insights courtesy of Jens-Uwe Mager and Christian Tismer who endorse the idea). When this succeeds we can start to code e.g. a POSIX-layer and wrap C-libaries (at runtime) from python. So much for now from my side. I am sure that as so many people inherit the good culture from c.l.py and python-dev we will have good discussions. Please note that there are some subscribers (e.g. Jens-Uwe Mager) who are experts but not neccessarily in core CPython. I hope/suggest that even naive sounding questions about Python and its core are welcomed. It certainly helps if more people understand the internals and the involved issues. I reserve the right to ask naive questions, myself :-) regards, holger krekel
In article <20030111214011.P1568@prim.han.de>, holger krekel <hpk@trillke.net> writes
Welcome to the Minimal Python list, ..... thanks for this
I hope/suggest that even naive sounding questions about Python and its core are welcomed. It certainly helps if more people understand the internals and the involved issues. I reserve the right to ask naive questions, myself :-)
regards,
holger krekel ..... A naive questions; hich platforms are developments first aimed at ie gnu/linux win32 etc?
Thomas Heller's ctypes module would seem to be a very good start at the generic C interface thing. It is pretty easy to use and although I don't have a complete grasp on it was a bit easier to use than calldll. I used both to do anygui interfaces to native windows api. Last time I discussed this with him Thomas said he was considering using the libffi library to do interfacing. I know that to be fairly portable. -- Robin Becker
[Robin Becker Sat, Jan 11, 2003 at 09:30:04PM +0000]
In article <20030111214011.P1568@prim.han.de>, holger krekel <hpk@trillke.net> writes
Welcome to the Minimal Python list, ..... thanks for this
I hope/suggest that even naive sounding questions about Python and its core are welcomed. It certainly helps if more people understand the internals and the involved issues. I reserve the right to ask naive questions, myself :-)
regards,
holger krekel ..... A naive questions; hich platforms are developments first aimed at ie gnu/linux win32 etc?
I guess it's going to be win32 & POSIX (MacOSX, and some linux/unix variants).
Thomas Heller's ctypes module would seem to be a very good start at the generic C interface thing. It is pretty easy to use and although I don't have a complete grasp on it was a bit easier to use than calldll. I used both to do anygui interfaces to native windows api.
I heard good things about it but it is currently a windows only solution, or not?
Last time I discussed this with him Thomas said he was considering using the libffi library to do interfacing. I know that to be fairly portable.
And that is a non-windows and non-mac solution? Anyway, both are interesting but actually Christian and others can probably judge better. This is obviously an issue for people knowing many different platforms. holger
holger krekel <hpk@trillke.net> writes:
[Robin Becker Sat, Jan 11, 2003 at 09:30:04PM +0000]
A naive questions; hich platforms are developments first aimed at ie gnu/linux win32 etc?
I guess it's going to be win32 & POSIX (MacOSX, and some linux/unix variants).
Thomas Heller's ctypes module would seem to be a very good start at the generic C interface thing. It is pretty easy to use and although I don't have a complete grasp on it was a bit easier to use than calldll. I used both to do anygui interfaces to native windows api.
I heard good things about it but it is currently a windows only solution, or not?
Last time I discussed this with him Thomas said he was considering using the libffi library to do interfacing. I know that to be fairly portable.
I've now a running version which uses libffi (except on Windows). Tested on an oldish SuSE 7.1 x86 system. The docs and readme's still need updating, but for the *impatient* I've uploaded a snapshot: http://starship.python.net/crew/theller/ctypes-0.3.5.tar.gz Thomas
On Mon, Jan 13, 2003 at 10:04:27AM +0100, Thomas Heller wrote:
I've now a running version which uses libffi (except on Windows). Tested on an oldish SuSE 7.1 x86 system.
The docs and readme's still need updating, but for the *impatient* I've uploaded a snapshot:
I just downloaded this version and I am trying to find a matching ffi library, which appears to be rather difficult. I am building on a gentoo system which has libffi 1.20 and I get problems with ffi_closure not being defined. Do I really need to check out the complete gcc source tree to get the proper libffi? -- Jens-Uwe Mager <pgp-mailto:62CFDB25>
Jens-Uwe Mager <jum@anubis.han.de> writes:
On Mon, Jan 13, 2003 at 10:04:27AM +0100, Thomas Heller wrote:
I've now a running version which uses libffi (except on Windows). Tested on an oldish SuSE 7.1 x86 system.
The docs and readme's still need updating, but for the *impatient* I've uploaded a snapshot:
I just downloaded this version and I am trying to find a matching ffi library, which appears to be rather difficult. I am building on a gentoo system which has libffi 1.20 and I get problems with ffi_closure not being defined. Do I really need to check out the complete gcc source tree to get the proper libffi?
I have two: The first one checked out via anonymous CVS following the instructions on the webpage http://sources.redhat.com/libfii/ . At least they updated the pointers on the webpage to be valid, which was not the case when I first got it. I then did ./configure make make install The configure command reported an error at the end: ./config.status: ./../config-ml.in: No such file or directory which I'm unable to resolve with my poor linux skills, but it works. The second one I got from Robin Becker, it seems to be a patched version of the first one, which includes further updates for Windows (stdcall and cdecl calling convention). He got it from somewhere else... Both should work, although I'm using the first one IIRC. I'm still using my own stuff on Windows, and libffi on the linux system. Unfortunately both have the filename libffi-2.00-beta.tar.gz. Should I upload a working tar.gz, or are these instructions sufficient? Thomas BTW: Is yours an x86 system?
The second one I got from Robin Becker, it seems to be a patched version of the first one, which includes further updates for Windows (stdcall and cdecl calling convention). He got it from somewhere else... The version I sent to Thomas came from Roger E Critchlow Jr <rec@elf.org> and was part of his Tcl ffidl package version 0.5. He was certainly into the foreign function interface thing in a big way and his package also included source for ffcall-1.6 http://www.gnu.org/directory/libs/c/ffcall.html and
In article <y95mtrq9.fsf@python.net>, Thomas Heller <theller@python.net> writes the platforms reported for that are Supported CPUs: (Put the GNU config.guess values here.) i386 i486-unknown-linux (gcc), i386-unknown-sysv4.0 (gcc, /usr/bin/cc, /usr/ucb/cc), i386-pc-solaris2.6 (gcc), i486-unknown-sco3.2v4.2 (gcc, cc -Of), i486-unknown-os2emx (gcc), i386-pc-cygwin32 (gcc), i386-pc-win32 (msvc4, msvc5) m68k m68k-next-nextstep3 (cc), m68k-sun-sunos4.0 (cc), m68k-unknown-linux (gcc) mips mips-sgi-irix4.0.5 (gcc, cc -ansi, cc -D__STDC__, cc -cckr), mips-sgi-irix5.2 (gcc, cc -ansi, cc -D__STDC__, cc -cckr), mips-sgi-irix5.3 (gcc, cc -ansi, cc -D__STDC__, cc -cckr), mips-sgi-irix6.2 (cc -32), mips-sgi-irix6.4 (cc -32, cc -n32, cc -64) sparc sparc-sun-sunos4.1.1 (gcc, cc), sparc-sun-solaris2.3 (gcc) sparc-sun-solaris2.4 (gcc, cc) alpha alpha-dec-osf3.0 (gcc, cc), alpha-dec-osf4.0 (gcc, cc) hppa hppa1.0-hp-hpux8.00 (gcc, cc), hppa1.1-hp-hpux9.05 (cc), hppa1.1-hp-hpux10.01 (cc), hppa2.0-hp-hpux10.20 (cc +DA1.1) arm -- untested rs6000 rs6000-ibm-aix3.2.5 (gcc, c89, xlc), powerpc-ibm-aix4.1.4.0 (cc) m88k -- untested convex -- untested -- Robin Becker
Robin Becker <robin@reportlab.com> writes:
In article <y95mtrq9.fsf@python.net>, Thomas Heller <theller@python.net> writes
The second one I got from Robin Becker, it seems to be a patched version of the first one, which includes further updates for Windows (stdcall and cdecl calling convention). He got it from somewhere else... The version I sent to Thomas came from Roger E Critchlow Jr <rec@elf.org> and was part of his Tcl ffidl package version 0.5.
Thanks for looking this up again, Robin.
He was certainly into the foreign function interface thing in a big way and his package also included source for ffcall-1.6 http://www.gnu.org/directory/libs/c/ffcall.html and the platforms reported for that are
I'm not keen on supporting ffcall, since it's GPL and so I cannot use it for the stuff I have to redistribute. OTOH I'm quite sure it would not be too difficult to support that, also, if useful. Thomas
In article <65sqcgxf.fsf@python.net>, Thomas Heller <theller@python.net> writes
Robin Becker <robin@reportlab.com> writes:
In article <y95mtrq9.fsf@python.net>, Thomas Heller <theller@python.net> writes
The second one I got from Robin Becker, it seems to be a patched version of the first one, which includes further updates for Windows (stdcall and ...... I'm not keen on supporting ffcall, since it's GPL and so I cannot use it for the stuff I have to redistribute. OTOH I'm quite sure it would not be too difficult to support that, also, if useful.
Thomas
No problem on that, I just wanted to mention some alternatives. -- Robin Becker
Robin Becker:
Thomas Heller's ctypes module would seem to be a very good start at the generic C interface thing. It is pretty easy to use and although I don't have a complete grasp on it was a bit easier to use than calldll. I used both to do anygui interfaces to native windows api.
Thomas Heller:
The docs and readme's still need updating, but for the *impatient* I've uploaded a snapshot:
Well, I had a look at the current version of ctypes, in the backgound of this project, of course. I only looked into _ctypes.c right now. callbacks.c and callproc.c have to be examined as well. My impressions are a bit mixed. First of all, this looks like a very complete module to manipulate every and all flavors of C types. I haven't played with it yet, what I promise to do. As an extension to the current, C based Python implementation, it is without doubt great stuff. There are lots of new object types together with lots of methods to create and manipulate them. What I liked really much (and I would probably like to steal this trick) is the most elegant implementation of alignment. Interested readers should read and understand lines 3309 ff of _ctypes.c . Now to the flipside. While this module is for sure of great use for CPython, I'm quite reluctant to use it as-is for Minimal Python. Reason? It is *way* too big, with 101 KB source and 4330 lines. While this is very ok with CPython, I think such a big module in C is exactly what we don't want for this project, since getting rid of large C sources is one of the first objectives of the project. This is absolutely not meant negatively to Thomas. Your module was not designed for this project. In the context of CPython, this is the way to do an efficient module. Everything has to be written down, in the flat way that C requires. That creates many source lines, many type denotations, and lots of similar looking methods. For Minimal Python, we need only a small percentage of this. We should pick some essential ideas and re-write that part in Python. We also only need some basic types to boot up the Python engine. Access to real memory can be provided by some functions in C, which should be mapped to opcodes of the virtual micro-machine. I'm anyway very thankful for this great resource. The most valuable thing are the ideas, which we should take into account. I'm also sure that we need to learn from the interfaces into library calls, pretty soon. Thanks a lot for this impressive implementation. Minimal Python will surely need some of the code and many of the ideas. cheers - chris
Christian Tismer <tismer@tismer.com> writes:
Robin Becker:
Thomas Heller's ctypes module would seem to be a very good start at the generic C interface thing. It is pretty easy to use and although I don't have a complete grasp on it was a bit easier to use than calldll. I used both to do anygui interfaces to native windows api.
Thomas Heller:
The docs and readme's still need updating, but for the *impatient* I've uploaded a snapshot:
Well, I had a look at the current version of ctypes, in the backgound of this project, of course. I only looked into _ctypes.c right now. callbacks.c and callproc.c have to be examined as well.
My impressions are a bit mixed.
First of all, this looks like a very complete module to manipulate every and all flavors of C types. I haven't played with it yet, what I promise to do.
As an extension to the current, C based Python implementation, it is without doubt great stuff. There are lots of new object types together with lots of methods to create and manipulate them.
What I liked really much (and I would probably like to steal this trick) is the most elegant implementation of alignment. Interested readers should read and understand lines 3309 ff of _ctypes.c .
Ahem, this code is nearly copied from Python's structmodule.c. So, this idea was someone else's.
Now to the flipside. While this module is for sure of great use for CPython, I'm quite reluctant to use it as-is for Minimal Python. Reason? It is *way* too big, with 101 KB source and 4330 lines.
While this is very ok with CPython, I think such a big module in C is exactly what we don't want for this project, since getting rid of large C sources is one of the first objectives of the project.
This is absolutely not meant negatively to Thomas. Your module was not designed for this project. In the context of CPython, this is the way to do an efficient module. Everything has to be written down, in the flat way that C requires. That creates many source lines, many type denotations, and lots of similar looking methods.
Yes, only counting the lines it is even larger than Jim Fulton's ExtensionClass.c - but the code is much less dense and easier to read, IMO. But don't take this wrong: I'm not going to argue with you that something else is needed for Minimal Python.
For Minimal Python, we need only a small percentage of this. We should pick some essential ideas and re-write that part in Python. We also only need some basic types to boot up the Python engine. Access to real memory can be provided by some functions in C, which should be mapped to opcodes of the virtual micro-machine.
Thanks for this great review, chris. Thomas
Thomas Heller wrote: ...
What I liked really much (and I would probably like to steal this trick) is the most elegant implementation of alignment. Interested readers should read and understand lines 3309 ff of _ctypes.c .
Ahem, this code is nearly copied from Python's structmodule.c. So, this idea was someone else's.
*blush* I've never completely read the structmodule source. Sorry. [talking about size]
Yes, only counting the lines it is even larger than Jim Fulton's ExtensionClass.c - but the code is much less dense and easier to read, IMO. But don't take this wrong: I'm not going to argue with you that something else is needed for Minimal Python.
Great! I was afraid you might take me wrong.
For Minimal Python, we need only a small percentage of this. We should pick some essential ideas and re-write that part in Python. We also only need some basic types to boot up the Python engine. Access to real memory can be provided by some functions in C, which should be mapped to opcodes of the virtual micro-machine.
I dislike the above paragraph now. I'm overloading the young project with demands and restrictions.
Thanks for this great review, chris.
After some sleep, I found this being not such a great review. I too much pointed on things which are only relevant through the nature of this project, and I argued with the final result in mind. Instead, I should have mentioned that it is pretty fine to use ctypes right now as it is, in order to get things flying. All considerations about re-coding things in Python don't apply to the early bootstrap phase. I-shouldn't-write-at-4-o'clock-in-the-night - chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
[holger krekel]
I think that - if in doubt - we follow the lead from CPython e.g. regarding coding style, language definition etc. Every deviation should be justified by the goals. I don't think we should follow a PEP process soon, though.
All seems reasonable. No need to stop using something that works and people are, in general, familiar with. Also would make it easier to apply any code from Mini-Py (or is there some already accepted abbreviation for this project?) back into CPython if so desired. <snip>
Bootstrapping will be a hot topic for Minimal Python itself. We probably first want to use the CPython development environemnt to get going. We probably want to base it on the python-2.3 cvs tree.
Once again I agree. I assume the other option is the maint-2.2 branch. That is not a good idea, in my opinion, since generators require a __future__ statement in 2.2. Might as well try to get all the good features we can into this. And speaking of features, how is the project going to view features that we all know are on their way out at some point? I am specifically thinking of integer division and old-style classes. Both are destined to disappear once Py3K comes out (whenever that is), and so should time be spend in implementing thess features? I am assuming backwards-compatibility takes precedence over code simplification but I thought I would double-check.
There is the idea of not using make/configure/automake but a simple understandable debuggable (read: python based) build environment. IOW words the number of dependencies for building Minimal Python should also be minimal.
What about A-A-P (http://www.a-a-p.org/index.html)? It's Python-based and seems like a well-designed tool (but I am a Vim fan so I am biased toward's the creator's work =). <snip> -Brett
holger krekel wrote:
Bootstrapping will be a hot topic for Minimal Python itself. We probably first want to use the CPython development environemnt to get going. We probably want to base it on the python-2.3 cvs tree.
There is the idea of not using make/configure/automake but a simple understandable debuggable (read: python based) build environment. IOW words the number of dependencies for building Minimal Python should also be minimal.
May I suggest scons (http://www.scons.org/)? It's what I would use in a new project today.
For avoiding the need of C-coded system modules there is interest to code a generalization of the "struct" module which allows *calling* functions at the C-level. This general C-"Extension" will be system-dependent. The gateway code between the "machine" and python probably needs to be coded in assembler: You can't "construct" a C-function call within ANSI-C. Both Microsoft and Apple have proprietary solutions for Java. (these insights courtesy of Jens-Uwe Mager and Christian Tismer who endorse the idea).
I will point people towards the code in Mozilla that does similar things, called xptcall: http://www.mozilla.org/scriptable/xptcall-faq.html It's XPCOM centered, but many of the ideas and the assembly code for mac, win32 and os2, and a bunch of unix variants (http://lxr.mozilla.org/mozilla/source/xpcom/reflect/xptcall/src/md/unix/) is there. Robin already mentioned libffi as well. Looking forward to participating in this very interesting project, -- david
There is the idea of not using make/configure/automake but a simple understandable debuggable (read: python based) build environment. IOW words the number of dependencies for building Minimal Python should also be minimal.
May I suggest scons (http://www.scons.org/)? It's what I would use in a new project today. SCons is a really nice tool, but what about the good old makefile (+ config.mk). We're talking about a small core and Python itself uses POSIX conforming source it shouldn't be that problem. And, SCons requires an installed Python. If an installed Python should be a requirement we could also try to use distutils (to have one dependency less).
Marc "Premature optimization is the root of all evil." -- Donald E. Knuth
[Marc Recht Sat, Jan 11, 2003 at 11:12:00PM +0100]
There is the idea of not using make/configure/automake but a simple understandable debuggable (read: python based) build environment. IOW words the number of dependencies for building Minimal Python should also be minimal.
May I suggest scons (http://www.scons.org/)? It's what I would use in a new project today. SCons is a really nice tool, but what about the good old makefile (+ config.mk). We're talking about a small core and Python itself uses POSIX conforming source it shouldn't be that problem. And, SCons requires an installed Python. If an installed Python should be a requirement we could also try to use distutils (to have one dependency less).
Using CPython as a first means of bootstrapping should only be there for the time beeing. It's simply practical to start with. Eventually there should be a self-contained bootstrapping process. IMO this hints at doing a very smallish VM which is able to drive the build process to a point where the next-stage VM with more features and a generic C-level interfacing could take over. SCons is certainly worth checking out but it might already use too many features of python to be used for initial bootstrapping. I doubt that distutils would be of much use at early stages. Eventually the requirement for CPython should go away. And once Minimal Python is unwrapped we don't need a build environment anymore :-) regards, holger
There is the idea of not using make/configure/automake Not using automake/autoconf is IMHO "The Right Thing".
but a simple understandable debuggable (read: python based) build environment. IOW words the number of dependencies for building Minimal Python should also be minimal. Why not using make ? It's installed on any system that has a c compiler, so it doesn't add another dependency. IMHO we could use it to bootstrap the core and then go on with something Python based (using the bootstrapped core).
While traveling through NetBSD's pkgsrc I came across this http://buildtool.sourceforge.net/ . I haven't tried it yet, but the description sounds promising. Marc "Premature optimization is the root of all evil." -- Donald E. Knuth
Marc Recht wrote:
There is the idea of not using make/configure/automake
Not using automake/autoconf is IMHO "The Right Thing".
but a simple understandable debuggable (read: python based) build environment. IOW words the number of dependencies for building Minimal Python should also be minimal.
Why not using make ? It's installed on any system that has a c compiler, so it doesn't add another dependency. IMHO we could use it to bootstrap the core and then go on with something Python based (using the bootstrapped core).
Make, Configure etc. aren't "really" portable to Windows. It's hard to know when you've written a portable makefile. Make doesn't scale. Make is error-prone, etc. There are lots of problems with Make, but this isn't really the place for that discussion. See pages like: http://sc-archive.codesourcery.com/sc_build http://sc-archive.codesourcery.com/entries/build/Machina/machina-appendix.ht... http://www.scons.org/doc/HTML/scons-python10/t1.html http://www.a-a-p.org/ --david
On Sat, Jan 11, 2003 at 09:40:11PM +0100, holger krekel wrote:
For avoiding the need of C-coded system modules there is interest to code a generalization of the "struct" module which allows *calling* functions at the C-level. This general C-"Extension" will be system-dependent.
Someone already mentioned Squeak, which I think is something everyone involved in this project should take a look at. (http://www.squeak.org/ ) I would also mention Gnu Smalltalk (http://www.gnu.org/software/smalltalk/); it has the best C-calling-from-interpreter interface I've ever seen. []s, |alo +---- -- Those who trade freedom for security lose both and deserve neither. -- http://www.laranja.org/ mailto:lalo@laranja.org pgp key: http://www.laranja.org/pessoal/pgp Eu jogo RPG! (I play RPG) http://www.eujogorpg.com.br/ GNU: never give up freedom http://www.gnu.org/
Lalo Martins <lalo@laranja.org> writes:
On Sat, Jan 11, 2003 at 09:40:11PM +0100, holger krekel wrote:
For avoiding the need of C-coded system modules there is interest to code a generalization of the "struct" module which allows *calling* functions at the C-level. This general C-"Extension" will be system-dependent.
Someone already mentioned Squeak, which I think is something everyone involved in this project should take a look at. (http://www.squeak.org/ )
I would also mention Gnu Smalltalk (http://www.gnu.org/software/smalltalk/); it has the best C-calling-from-interpreter interface I've ever seen.
I took some minutes to look over the manual, expecially http://www.gnu.org/software/smalltalk/gst-manual/gst_21.html To be honest, in spirit it looks very similar to what I implemented in the Python ctypes module. Except, that you cannot store functions pointers into CObject instances in GNU smalltalk (at least it seems so from this web-page), but in ctypes. Thomas
participants (9)
-
Brett Cannon -
Christian Tismer -
David Ascher -
holger krekel -
Jens-Uwe Mager -
Lalo Martins -
Marc Recht -
Robin Becker -
Thomas Heller