I would like to run Python scripts on an embedded MIPS Linux platform having only 2 MiB of flash ROM and 16 MiB of RAM for everything. Current (2.5) stripped and gzipped (I am going to use a compressed filesystem) CPython binary, compiled with defaults on a i386/glibc Linux, results in 500 KiB of "flash". How to make the Python interpreter even smaller? - can I completely drop out lexical analysis of sourcecode and compilation to bytecode? is it relevant enough to the size of interpreter? - should I drop "useless" compiled-in modules? (what I need is a replacement for advanced bash scripting, being able to write more complex scripts and avoid forking tens of processes for things like searching filesystem, formating dates etc.) I don't want to re-invent the wheel, but all my attempts at finding Python for embedded systems ended in instructions for embedding Python in another program :-) Can you give me any information to start with? I would prefer stripping current version of Python rather than returning to a years-old (but smaller) version and remembering what of the new syntax/functionality to avoid. TIA, Milan
Milan Krcmar wrote:
Current (2.5) stripped and gzipped (I am going to use a compressed filesystem) CPython binary, compiled with defaults on a i386/glibc Linux, results in 500 KiB of "flash". How to make the Python interpreter even smaller?
In my experience, the biggest gain can be obtained by dropping the rarely-used CJK codecs (for Asian languages). That should sum up to almost 800K (uncompressed), IIRC. After that, I once had to strip down the binary even more, and found out (by guesswork and inspection of map files) that there is no other low hanging fruit. By carefully selecting which modules to link in, I was able to reduce of another 300K or so, but nothing really incredible. I would also suggest -ffunction-sections in these cases, but you might already know that. Giovanni Bajo
Milan Krcmar <krcmar@datinel.cz> writes:
I would like to run Python scripts on an embedded MIPS Linux platform having only 2 MiB of flash ROM and 16 MiB of RAM for everything.
Current (2.5) stripped and gzipped (I am going to use a compressed filesystem) CPython binary, compiled with defaults on a i386/glibc Linux, results in 500 KiB of "flash". How to make the Python interpreter even smaller?
- can I completely drop out lexical analysis of sourcecode and compilation to bytecode? is it relevant enough to the size of interpreter?
I don't think there's an configure flag for this or anything, and it might be a bit hairy to do it, but it's possible and it would probably save a bit. There is a configure option to remove unicode support. It's not terribly well supported and stops working every now and again, but it's probably much easier to start with. There was at one point and may still be an option to not include the complex type.
- should I drop "useless" compiled-in modules? (what I need is a replacement for advanced bash scripting, being able to write more complex scripts and avoid forking tens of processes for things like searching filesystem, formating dates etc.)
Yes, definitely.
I don't want to re-invent the wheel, but all my attempts at finding Python for embedded systems ended in instructions for embedding Python in another program :-)
Can you give me any information to start with? I would prefer stripping current version of Python rather than returning to a years-old (but smaller) version and remembering what of the new syntax/functionality to avoid.
Well, I would start by looking at what is taking up the space... Cheers, mwh -- C++ is a siren song. It *looks* like a HLL in which you ought to be able to write an application, but it really isn't. -- Alain Picard, comp.lang.lisp
Milan Krcmar schrieb:
Can you give me any information to start with? I would prefer stripping current version of Python rather than returning to a years-old (but smaller) version and remembering what of the new syntax/functionality to avoid.
I would start with dropping support for dynamic loading of extension modules, and link all necessary modules statically. Then, do what Michael Hudson says: find out what is taking up space. size */*.o|sort -n should give a good starting point; on my system, I get [...] 29356 1416 156 30928 78d0 Objects/classobject.o 30663 0 0 30663 77c7 Objects/unicodectype.o 33530 480 536 34546 86f2 Python/Python-ast.o 33624 1792 616 36032 8cc0 Objects/longobject.o 36603 16 288 36907 902b Python/ceval.o 36710 2532 0 39242 994a Modules/_sre.o 39169 9473 1032 49674 c20a Objects/stringobject.o 52965 0 36 53001 cf09 Python/compile.o 66197 4592 436 71225 11639 Objects/typeobject.o 74111 9779 1160 85050 14c3a Objects/unicodeobject.o Michael already mentioned you can drop unicodeobject if you want to. compile.o would also offer savings, but stripping it might not be easy. Dropping _sre is quite easy. If you manage to drop compile.o, then dropping Python-ast.o (along with the rest of the compiler) should also be possible. unicodectype will go away if the Unicode type goes, but can probably be removed separately. And so on. When you come to a solution that satisfies your needs, don't forget to document it somewhere. Regards, Martin
Thank you people. I'm going to try to strip unneeded things and let you know the result. Along with running Python on an embedded system, I am considering two more things. Suppose the system to be a small Linux router, which, after the kernel starts, merely configures lots of parameters of the kernel and then runs some daemons for gathering statistics and allowing remote control of the host. Python helps mainly in the startup phase of configuring kernel according to a human-readable confgiuration files. This has been solved by shell scripts. Python is not as suitable for running external processes and process pipes as a shell, but I'd like to write a module (at least) helping him in the sense of scsh (a "Scheme shell", http://www.scsh.net). A more advanced solution is to replace system's init (/sbin/init) by Python. It should even speed the startup up as it will not need to run shell many times. To avoid running another processes, I want to "port them" to Python. Processes for kernel configuration, like iproute2, iptables etc. are often built above its own library, which can be used as a start point. (Yes, it does matter, at startup, routers run such processes hundreds times). Milan On Sun, Sep 24, 2006 at 06:49:34AM +0200, "Martin v. Löwis" wrote:
Milan Krcmar schrieb:
Can you give me any information to start with? I would prefer stripping current version of Python rather than returning to a years-old (but smaller) version and remembering what of the new syntax/functionality to avoid.
I would start with dropping support for dynamic loading of extension modules, and link all necessary modules statically.
Then, do what Michael Hudson says: find out what is taking up space.
size */*.o|sort -n
should give a good starting point; on my system, I get
[...] 29356 1416 156 30928 78d0 Objects/classobject.o 30663 0 0 30663 77c7 Objects/unicodectype.o 33530 480 536 34546 86f2 Python/Python-ast.o 33624 1792 616 36032 8cc0 Objects/longobject.o 36603 16 288 36907 902b Python/ceval.o 36710 2532 0 39242 994a Modules/_sre.o 39169 9473 1032 49674 c20a Objects/stringobject.o 52965 0 36 53001 cf09 Python/compile.o 66197 4592 436 71225 11639 Objects/typeobject.o 74111 9779 1160 85050 14c3a Objects/unicodeobject.o
Michael already mentioned you can drop unicodeobject if you want to. compile.o would also offer savings, but stripping it might not be easy. Dropping _sre is quite easy. If you manage to drop compile.o, then dropping Python-ast.o (along with the rest of the compiler) should also be possible. unicodectype will go away if the Unicode type goes, but can probably be removed separately. And so on.
When you come to a solution that satisfies your needs, don't forget to document it somewhere.
Regards, Martin
Milan Krcmar wrote:
Thank you people. I'm going to try to strip unneeded things and let you know the result.
Along with running Python on an embedded system, I am considering two more things. Suppose the system to be a small Linux router, which, after the kernel starts, merely configures lots of parameters of the kernel and then runs some daemons for gathering statistics and allowing remote control of the host.
Python helps mainly in the startup phase of configuring kernel according to a human-readable confgiuration files. This has been solved by shell scripts. Python is not as suitable for running external processes and process pipes as a shell, but I'd like to write a module (at least) helping him in the sense of scsh (a "Scheme shell", http://www.scsh.net).
A more advanced solution is to replace system's init (/sbin/init) by Python. It should even speed the startup up as it will not need to run shell many times. To avoid running another processes, I want to "port them" to Python. Processes for kernel configuration, like iproute2, iptables etc. are often built above its own library, which can be used as a start point. (Yes, it does matter, at startup, routers run such processes hundreds times).
Milan
One alternative you might want to look into is the language "Lua" (www.lua.org), which is similar to Python in some respects (also has some similarities to Javascript), but specifically optimized for embedding in larger apps - meaning that it has a much smaller footprint, a much smaller standard library, less built-in data types and so on. (For example, dicts, lists, and objects are all merged into a single type called a 'table', which is just a generic indexable container.) Lua's C API consists of just a few dozen functions. It's not as powerful as Python of course, although it's surprisingly powerful for its size - it has closures, continuations, and all of the goodness you would expect from a modern language. Lua provides 'meta-mechanisms' for extending the language rather than implementing language features directly. So even though it's not a pure object-oriented language, it provides mechanisms for implementing classes and inheritance. And it's fast, since it has less baggage to carry around. It has a few warts - for example, I don't like the fact that referring to an undefined variable silently returns nil instead of returning an error, but I suppose in some environments that's a feature. A lot of game companies use Lua for embedded scripting languages in their games. (Console-based games in particular have strict memory requirements, since there's no virtual memory on consoles.) -- Talin
I would like to run Python scripts on an embedded MIPS Linux platform having only 2 MiB of flash ROM and 16 MiB of RAM for everything. (...)
Have you looked at Python for S60 and Python for the Maemo platform? If not directly useful, they should provide some hints. [1] http://opensource.nokia.com/projects/pythonfors60/ [2] http://pymaemo.sf.net -- Gustavo Niemeyer http://niemeyer.net
participants (6)
-
"Martin v. Löwis" -
Giovanni Bajo -
Gustavo Niemeyer -
Michael Hudson -
Milan Krcmar -
Talin