How to customize CPython to a minimal set
Hi, all There is a request to run python in a Linux-based embedded resource constrained system with sqlite3 support. So many features are not required, like posixmodule, signalmodule, hashtable ... But seems there are some dependencies among the Modules/Parser/Python/Objects/Programs... Is there a way to tailor CPython 3 to a minimal set with sqlite3 (the less syscalls the better) ? Is it possible to do that? Thank you.
Have you considered starting with micropython? It’s made for embedded systems and fully supports Python 3 syntax. Adding sqlite3 support to it will be less work than stripping all the I/O from CPython. —Guido On Mon, Jul 20, 2020 at 06:48 Huang, Yang <yang.huang@intel.com> wrote:
Hi, all
There is a request to run python in a Linux-based embedded resource constrained system with sqlite3 support.
So many features are not required, like posixmodule, signalmodule, hashtable ... But seems there are some dependencies among the Modules/Parser/Python/Objects/Programs...
Is there a way to tailor CPython 3 to a minimal set with sqlite3 (the less syscalls the better) ? Is it possible to do that?
Thank you. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ECPLKXQ4... Code of Conduct: http://python.org/psf/codeofconduct/
-- --Guido (mobile)
Hi, Guido Yes. Micropyhton is also in consideration. But sqlite3 is the first usage. There should be some additional features like numpy, scipy... Not sure if micropython supports well? Or is there a feasible way to strip CPython ? Thanks. From: Guido van Rossum <guido@python.org> Sent: Monday, July 20, 2020 10:45 PM To: Huang, Yang <yang.huang@intel.com> Cc: python-dev@python.org Subject: Re: [Python-Dev] How to customize CPython to a minimal set Have you considered starting with micropython? It’s made for embedded systems and fully supports Python 3 syntax. Adding sqlite3 support to it will be less work than stripping all the I/O from CPython. —Guido On Mon, Jul 20, 2020 at 06:48 Huang, Yang <yang.huang@intel.com<mailto:yang.huang@intel.com>> wrote: Hi, all There is a request to run python in a Linux-based embedded resource constrained system with sqlite3 support. So many features are not required, like posixmodule, signalmodule, hashtable ... But seems there are some dependencies among the Modules/Parser/Python/Objects/Programs... Is there a way to tailor CPython 3 to a minimal set with sqlite3 (the less syscalls the better) ? Is it possible to do that? Thank you. _______________________________________________ Python-Dev mailing list -- python-dev@python.org<mailto:python-dev@python.org> To unsubscribe send an email to python-dev-leave@python.org<mailto:python-dev-leave@python.org> https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ECPLKXQ4... Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido (mobile)
I expect it will be unfeasible to strip CPython. If you disagree, try it. ;-) On Mon, Jul 20, 2020 at 22:35 Huang, Yang <yang.huang@intel.com> wrote:
Hi, Guido
Yes. Micropyhton is also in consideration.
But sqlite3 is the first usage. There should be some additional features like numpy, scipy... Not sure if micropython supports well?
Or is there a feasible way to strip CPython ?
Thanks.
*From:* Guido van Rossum <guido@python.org> *Sent:* Monday, July 20, 2020 10:45 PM *To:* Huang, Yang <yang.huang@intel.com> *Cc:* python-dev@python.org *Subject:* Re: [Python-Dev] How to customize CPython to a minimal set
Have you considered starting with micropython? It’s made for embedded systems and fully supports Python 3 syntax. Adding sqlite3 support to it will be less work than stripping all the I/O from CPython.
—Guido
On Mon, Jul 20, 2020 at 06:48 Huang, Yang <yang.huang@intel.com> wrote:
Hi, all
There is a request to run python in a Linux-based embedded resource constrained system with sqlite3 support.
So many features are not required, like posixmodule, signalmodule, hashtable ... But seems there are some dependencies among the Modules/Parser/Python/Objects/Programs...
Is there a way to tailor CPython 3 to a minimal set with sqlite3 (the less syscalls the better) ? Is it possible to do that?
Thank you. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ECPLKXQ4... Code of Conduct: http://python.org/psf/codeofconduct/
--
--Guido (mobile)
-- --Guido (mobile)
Thank you for all your comments. I cannot agree any more. I did try but there were so many dependencies. One change all change. Micropython is a choice. But not sure if numpy and sqlite3 can be supported well. And what’s the compatibility of the libs in Pypi. From: Guido van Rossum <guido@python.org> Sent: Tuesday, July 21, 2020 10:57 PM To: Huang, Yang <yang.huang@intel.com> Cc: python-dev@python.org Subject: Re: [Python-Dev] How to customize CPython to a minimal set I expect it will be unfeasible to strip CPython. If you disagree, try it. ;-) On Mon, Jul 20, 2020 at 22:35 Huang, Yang <yang.huang@intel.com<mailto:yang.huang@intel.com>> wrote: Hi, Guido Yes. Micropyhton is also in consideration. But sqlite3 is the first usage. There should be some additional features like numpy, scipy... Not sure if micropython supports well? Or is there a feasible way to strip CPython ? Thanks. From: Guido van Rossum <guido@python.org<mailto:guido@python.org>> Sent: Monday, July 20, 2020 10:45 PM To: Huang, Yang <yang.huang@intel.com<mailto:yang.huang@intel.com>> Cc: python-dev@python.org<mailto:python-dev@python.org> Subject: Re: [Python-Dev] How to customize CPython to a minimal set Have you considered starting with micropython? It’s made for embedded systems and fully supports Python 3 syntax. Adding sqlite3 support to it will be less work than stripping all the I/O from CPython. —Guido On Mon, Jul 20, 2020 at 06:48 Huang, Yang <yang.huang@intel.com<mailto:yang.huang@intel.com>> wrote: Hi, all There is a request to run python in a Linux-based embedded resource constrained system with sqlite3 support. So many features are not required, like posixmodule, signalmodule, hashtable ... But seems there are some dependencies among the Modules/Parser/Python/Objects/Programs... Is there a way to tailor CPython 3 to a minimal set with sqlite3 (the less syscalls the better) ? Is it possible to do that? Thank you. _______________________________________________ Python-Dev mailing list -- python-dev@python.org<mailto:python-dev@python.org> To unsubscribe send an email to python-dev-leave@python.org<mailto:python-dev-leave@python.org> https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ECPLKXQ4... Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido (mobile) -- --Guido (mobile)
Sadly micropython is not intended to support numerical libraries and other such complex modules: the support for the Python standard library is pretty much non-existent. Kind regards, Steve On Wed, Jul 22, 2020 at 3:10 PM Huang, Yang <yang.huang@intel.com> wrote:
Thank you for all your comments.
I cannot agree any more. I did try but there were so many dependencies. One change all change.
Micropython is a choice. But not sure if numpy and sqlite3 can be supported well. And what’s the compatibility of the libs in Pypi.
*From:* Guido van Rossum <guido@python.org> *Sent:* Tuesday, July 21, 2020 10:57 PM *To:* Huang, Yang <yang.huang@intel.com> *Cc:* python-dev@python.org *Subject:* Re: [Python-Dev] How to customize CPython to a minimal set
I expect it will be unfeasible to strip CPython. If you disagree, try it. ;-)
On Mon, Jul 20, 2020 at 22:35 Huang, Yang <yang.huang@intel.com> wrote:
Hi, Guido
Yes. Micropyhton is also in consideration.
But sqlite3 is the first usage. There should be some additional features like numpy, scipy... Not sure if micropython supports well?
Or is there a feasible way to strip CPython ?
Thanks.
*From:* Guido van Rossum <guido@python.org> *Sent:* Monday, July 20, 2020 10:45 PM *To:* Huang, Yang <yang.huang@intel.com> *Cc:* python-dev@python.org *Subject:* Re: [Python-Dev] How to customize CPython to a minimal set
Have you considered starting with micropython? It’s made for embedded systems and fully supports Python 3 syntax. Adding sqlite3 support to it will be less work than stripping all the I/O from CPython.
—Guido
On Mon, Jul 20, 2020 at 06:48 Huang, Yang <yang.huang@intel.com> wrote:
Hi, all
There is a request to run python in a Linux-based embedded resource constrained system with sqlite3 support.
So many features are not required, like posixmodule, signalmodule, hashtable ... But seems there are some dependencies among the Modules/Parser/Python/Objects/Programs...
Is there a way to tailor CPython 3 to a minimal set with sqlite3 (the less syscalls the better) ? Is it possible to do that?
Thank you. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ECPLKXQ4... Code of Conduct: http://python.org/psf/codeofconduct/
--
--Guido (mobile)
--
--Guido (mobile) _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OMUGASSN... Code of Conduct: http://python.org/psf/codeofconduct/
I don't know if they suit your purposes, but both sqlite3 and a small array implementation are already available: https://github.com/micropython/micropython-lib/tree/master/sqlite3 https://github.com/v923z/micropython-ulab
There are tools in the python ecosystem that know how to build stand alone apps for python. These tools can figure out the minimum set of python and .so/.dylib/.dll that are needed. I use these tools when building apps for Windows and macOS where I do not require that python is pre-installed. You can use the same technics for unix systems. 1. Start with standard python 2. pip install deps 3. run tool to create minimum app If you build multiple apps then you can share common files to cut down size as well. Barry
On 22 Jul 2020, at 14:31, Huang, Yang <yang.huang@intel.com> wrote:
Thank you for all your comments. I cannot agree any more. I did try but there were so many dependencies. One change all change.
Micropython is a choice. But not sure if numpy and sqlite3 can be supported well. And what’s the compatibility of the libs in Pypi.
From: Guido van Rossum <guido@python.org <mailto:guido@python.org>> Sent: Tuesday, July 21, 2020 10:57 PM To: Huang, Yang <yang.huang@intel.com <mailto:yang.huang@intel.com>> Cc: python-dev@python.org <mailto:python-dev@python.org> Subject: Re: [Python-Dev] How to customize CPython to a minimal set
I expect it will be unfeasible to strip CPython. If you disagree, try it. ;-)
On Mon, Jul 20, 2020 at 22:35 Huang, Yang <yang.huang@intel.com <mailto:yang.huang@intel.com>> wrote: Hi, Guido
Yes. Micropyhton is also in consideration. But sqlite3 is the first usage. There should be some additional features like numpy, scipy... Not sure if micropython supports well?
Or is there a feasible way to strip CPython ?
Thanks.
From: Guido van Rossum <guido@python.org <mailto:guido@python.org>> Sent: Monday, July 20, 2020 10:45 PM To: Huang, Yang <yang.huang@intel.com <mailto:yang.huang@intel.com>> Cc: python-dev@python.org <mailto:python-dev@python.org> Subject: Re: [Python-Dev] How to customize CPython to a minimal set
Have you considered starting with micropython? It’s made for embedded systems and fully supports Python 3 syntax. Adding sqlite3 support to it will be less work than stripping all the I/O from CPython.
—Guido
On Mon, Jul 20, 2020 at 06:48 Huang, Yang <yang.huang@intel.com <mailto:yang.huang@intel.com>> wrote:
Hi, all
There is a request to run python in a Linux-based embedded resource constrained system with sqlite3 support.
So many features are not required, like posixmodule, signalmodule, hashtable ... But seems there are some dependencies among the Modules/Parser/Python/Objects/Programs...
Is there a way to tailor CPython 3 to a minimal set with sqlite3 (the less syscalls the better) ? Is it possible to do that?
Thank you. _______________________________________________ Python-Dev mailing list -- python-dev@python.org <mailto:python-dev@python.org> To unsubscribe send an email to python-dev-leave@python.org <mailto:python-dev-leave@python.org> https://mail.python.org/mailman3/lists/python-dev.python.org/ <https://mail.python.org/mailman3/lists/python-dev.python.org/> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ECPLKXQ4... <https://mail.python.org/archives/list/python-dev@python.org/message/ECPLKXQ4...> Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/> -- --Guido (mobile) -- --Guido (mobile) _______________________________________________ Python-Dev mailing list -- python-dev@python.org <mailto:python-dev@python.org> To unsubscribe send an email to python-dev-leave@python.org <mailto:python-dev-leave@python.org> https://mail.python.org/mailman3/lists/python-dev.python.org/ <https://mail.python.org/mailman3/lists/python-dev.python.org/> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OMUGASSN... <https://mail.python.org/archives/list/python-dev@python.org/message/OMUGASSN...> Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/>
On 2020-07-21 06:33, Huang, Yang wrote:
Hi, Guido
Yes. Micropyhton is also in consideration.
But sqlite3 is the first usage. There should be some additional features like numpy, scipy... Not sure if micropython supports well?
Or is there a feasible way to strip CPython ?
Thanks.
You want a minimal Python, yet with numpy, scipy, ...? Those sound like contradictory goals to me!
*From:* Guido van Rossum <guido@python.org> *Sent:* Monday, July 20, 2020 10:45 PM *To:* Huang, Yang <yang.huang@intel.com> *Cc:* python-dev@python.org *Subject:* Re: [Python-Dev] How to customize CPython to a minimal set
Have you considered starting with micropython? It’s made for embedded systems and fully supports Python 3 syntax. Adding sqlite3 support to it will be less work than stripping all the I/O from CPython.
—Guido
On Mon, Jul 20, 2020 at 06:48 Huang, Yang <yang.huang@intel.com <mailto:yang.huang@intel.com>> wrote:
Hi, all
There is a request to run python in a Linux-based embedded resource constrained system with sqlite3 support.
So many features are not required, like posixmodule, signalmodule, hashtable ... But seems there are some dependencies among the Modules/Parser/Python/Objects/Programs...
Is there a way to tailor CPython 3 to a minimal set with sqlite3 (the less syscalls the better) ? Is it possible to do that?
Thank you.
On 21Jul2020 0633, Huang, Yang wrote:
Yes. Micropyhton is also in consideration.
But sqlite3 is the first usage. There should be some additional features like numpy, scipy... Not sure if micropython supports well?
Or is there a feasible way to strip CPython ?
Only by manually removing modules from your own build. However, if you do that but want to continue using third-party packages, you'll find that there is very little you can remove (and nothing of significance). The most interesting modules to omit are ssl and ctypes, because of the exposure to likely security issues (via OpenSSL and libffi), but practically every module out there will have some sort of dependency on these. So you'll end up only removing small/niche modules that might save you a couple of kilobytes of Python code. Of the modules you suggested initially, I think you'll find that you need all of them for SQLite, let alone more complex libraries such as numpy. In theory, we could have a smaller set of modules, but in practice there are so many cross-dependencies already out there that there isn't much to gain right now. It would take a bit of an ecosystem "reset" (i.e. develop completely new popular libraries with more narrowly scoped dependencies) to narrow things down at this point, and there's not much chance of that happening. Cheers, Steve
On 7/20/20 10:30 AM, Huang, Yang wrote:
Hi, all
There is a request to run python in a Linux-based embedded resource constrained system with sqlite3 support.
So many features are not required, like posixmodule, signalmodule, hashtable ... But seems there are some dependencies among the Modules/Parser/Python/Objects/Programs...
Is there a way to tailor CPython 3 to a minimal set with sqlite3 (the less syscalls the better) ? Is it possible to do that?
CPython comes with the promise of all batteries included, however sometimes it feels like a complete power plant. For packaging purposes, most Linux distros make the decision to ship CPython as a set of runtime packages, and a set of packages used for development (and building C extensions). Breaking that down for the Debian/Ubuntu packages comes down to these sizes (unpacked, uncompressed, on x86_64): minimal: 5516K debian/libpython3.9-minimal 5856K debian/python3.9-minimal stdlib: 8528K debian/libpython3.9-stdlib 624K debian/python3.9 development: 19468K debian/libpython3.9-dev 25804K debian/libpython3.9-testsuite 1232K debian/python3-distutils 668K debian/python3-lib2to3 548K debian/python3.9-dev extra modules: 1648K debian/idle-python3.9 5208K debian/python3.9-examples 132K debian/python3-gdbm 844K debian/python3-tk What you don't see from the sizes, are the extra dependencies which add to the size, e.g. the X stack for tk, or readline/curses/crypto for stdlib. The "minimal" set may sound nice, however it's not used in practice, because the set is not well defined, and it's difficult to keep the minimal set as a self-contained set of stdlib modules, and the minimal usually keeps growing, never shrinks. It looks like this "minimal" set is even too much for your purposed, so you would have to scale down this set even further. Matthias
participants (8)
-
Barry Scott -
Guido van Rossum -
Huang, Yang -
Matthias Klose -
MRAB -
Simon Cross -
Steve Dower -
Steve Holden