Calling python from C completely statically
Hi, I've been working for a while on having the entire python interpreter with all his modules statically linked in a binary that can be called with arbitrary argument which will be passed to the python interpreter. I've been able to statically compile Python in a single binary with no dependencies for Python2.7 and using using this code to call the interpreter with arbitrary code: #define Py_NO_ENABLE_SHARED #include <Python.h> int main(int argc, char** argv) { Py_NoSiteFlag = 1; Py_InitializeEx(0); PyRun_SimpleString(argv[1]); Py_Finalize(); } This program compiles and does not rely in any dependency but the "non frozen" (.py) modules are not loaded so when I do a simple `import random` I get: Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: No module named random I've created a bug issue (https://bugs.python.org/issue34057) speaking about it and Nick Colghlan pointed: " cx_freeze is an illustrative example to look at in that regard, as it preconfigures the interpreter to be able to find the cx_freeze generated zip archive that has the program's Python modules in it: https://github.com/anthony-tuininga/cx_Freeze/blob/master/source/bases/Commo... The technique that cx_freeze doesn't use yet is to combine the statically linked Python binary and the generated zip archive into a single file (similar to what zipapp does), and adjust the sys.path definition inside the binary to refer back to the executable itself (since executable files can have arbitrary content appended, while zip files can have arbitrary content *pre*pended). " As I understand, cx_freeze creates a zip file with the dependencies for a specific python code and creates a zip with them. My question is, could I create a zip file with every standard module (plus some extra that I may install with pip) and using that zip file from C? If so, how can I do that? I'm happy to work on this if I get indications. BTW: I do not want to convert a python code snippet to exe or anything like that but calling python from C in an absolutely standalone version. Thank you -- Alberto García Illera GPG Public Key: https://goo.gl/twKUUv
On 8 July 2018 at 09:01, Alberto Garcia <agarciaillera@gmail.com> wrote:
Hi,
I've been working for a while on having the entire python interpreter with all his modules statically linked in a binary that can be called with arbitrary argument which will be passed to the python interpreter.
I've been able to statically compile Python in a single binary with no dependencies for Python2.7 and using using this code to call the interpreter with arbitrary code:
#define Py_NO_ENABLE_SHARED #include <Python.h>
int main(int argc, char** argv) { Py_NoSiteFlag = 1; Py_InitializeEx(0); PyRun_SimpleString(argv[1]); Py_Finalize(); }
This program compiles and does not rely in any dependency but the "non frozen" (.py) modules are not loaded so when I do a simple `import random` I get: Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: No module named random
I've created a bug issue (https://bugs.python.org/issue34057) speaking about it and Nick Colghlan pointed:
" cx_freeze is an illustrative example to look at in that regard, as it preconfigures the interpreter to be able to find the cx_freeze generated zip archive that has the program's Python modules in it: https://github.com/anthony-tuininga/cx_Freeze/blob/master/source/bases/Commo...
The technique that cx_freeze doesn't use yet is to combine the statically linked Python binary and the generated zip archive into a single file (similar to what zipapp does), and adjust the sys.path definition inside the binary to refer back to the executable itself (since executable files can have arbitrary content appended, while zip files can have arbitrary content *pre*pended). "
As I understand, cx_freeze creates a zip file with the dependencies for a specific python code and creates a zip with them. My question is, could I create a zip file with every standard module (plus some extra that I may install with pip) and using that zip file from C? If so, how can I do that?
I'm happy to work on this if I get indications.
BTW: I do not want to convert a python code snippet to exe or anything like that but calling python from C in an absolutely standalone version.
This question is probably more appropriate for python-list, but yes, you certainly can do this. The "Embedded" distributions of Python for Windows essentially do this already. IIRC, they are only available for Python 3.x, so you may find you have some hurdles to overcome if you have to remain on Python 2.7, but in principle it's possible. One further point you may need to consider - a proportion of the standard library is in the form of shared C extensions (PYDs in Windows - I don't know if you're using Windows or Unix from what you say above). You can't (without significant extra work) load a binary extension direct from a zip file, so you'll need to either do that extra work (which is platform specific and fragile, I believe) or be prepared to ship supporting DLLs alongside your application (this is the approach the embedded distribution takes). Hope this helps, Paul
On 8 July 2018 at 21:34, Paul Moore <p.f.moore@gmail.com> wrote:
This question is probably more appropriate for python-list, but yes, you certainly can do this. The "Embedded" distributions of Python for Windows essentially do this already. IIRC, they are only available for Python 3.x, so you may find you have some hurdles to overcome if you have to remain on Python 2.7, but in principle it's possible.
One further point you may need to consider - a proportion of the standard library is in the form of shared C extensions (PYDs in Windows - I don't know if you're using Windows or Unix from what you say above). You can't (without significant extra work) load a binary extension direct from a zip file, so you'll need to either do that extra work (which is platform specific and fragile, I believe) or be prepared to ship supporting DLLs alongside your application (this is the approach the embedded distribution takes).
That's the part of the problem that Alberto's static linking solves - all of the standard library's extension modules are top level ones (at least as far as I am aware), so we support building them as statically linked builtin modules instead (we just don't do it routinely, because we don't have any great desire to make the main executable even larger than it already is). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Hey there, Yes, the part of having the pyd modules built in in library is already done. I followed the instructions in the README. What I would like to know now is how to embed the non frozen python (py) modules. Can you guys please point me in the right direction. Thank you On Sun, Jul 8, 2018 at 7:07 AM Nick Coghlan <ncoghlan@gmail.com> wrote:
On 8 July 2018 at 21:34, Paul Moore <p.f.moore@gmail.com> wrote:
This question is probably more appropriate for python-list, but yes, you certainly can do this. The "Embedded" distributions of Python for Windows essentially do this already. IIRC, they are only available for Python 3.x, so you may find you have some hurdles to overcome if you have to remain on Python 2.7, but in principle it's possible.
One further point you may need to consider - a proportion of the standard library is in the form of shared C extensions (PYDs in Windows - I don't know if you're using Windows or Unix from what you say above). You can't (without significant extra work) load a binary extension direct from a zip file, so you'll need to either do that extra work (which is platform specific and fragile, I believe) or be prepared to ship supporting DLLs alongside your application (this is the approach the embedded distribution takes).
That's the part of the problem that Alberto's static linking solves - all of the standard library's extension modules are top level ones (at least as far as I am aware), so we support building them as statically linked builtin modules instead (we just don't do it routinely, because we don't have any great desire to make the main executable even larger than it already is).
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
-- Alberto García Illera GPG Public Key: https://goo.gl/twKUUv
On 9 July 2018 at 03:10, Alberto Garcia <agarciaillera@gmail.com> wrote:
Hey there,
Yes, the part of having the pyd modules built in in library is already done. I followed the instructions in the README. What I would like to know now is how to embed the non frozen python (py) modules. Can you guys please point me in the right direction.
The gist is to: 1. take the entire Lib directory and put it in a zip archive 2. use the approach demonstrated in cx_freeze to point sys.path in your static executable at that zip archive 3. adjust your C code to point sys.path back at the executable itself, and then combine your executable and the zip archive into a single contiguous file (similar to what zipapp does with it's helper script and app archive) There are likely to still be rough edges when doing that, since this isn't a well tested configuration. When all else fails, find the part of the source code responsible for any error messages you're seeing, and try to work out if there's a setting you can tweak to avoid hitting that code path. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Thank you for your response, I was thinking on creating that zip file with the content of the Lib folder and having my c code to download it over the network and have it in memory. I guess that the zip file should have no compression at all right? When you say that I need to use the cx_freeze approach what do you mean? Can you point me to where they do that? And why changing sys.path again to the executable again? Which part of the executable? I'll put my efforts in this. Thank you On Mon, Jul 9, 2018 at 7:16 AM Nick Coghlan <ncoghlan@gmail.com> wrote:
Hey there,
Yes, the part of having the pyd modules built in in library is already done. I followed the instructions in the README. What I would like to know now is how to embed the non frozen python (py) modules. Can you guys please
On 9 July 2018 at 03:10, Alberto Garcia <agarciaillera@gmail.com> wrote: point
me in the right direction.
The gist is to:
1. take the entire Lib directory and put it in a zip archive 2. use the approach demonstrated in cx_freeze to point sys.path in your static executable at that zip archive 3. adjust your C code to point sys.path back at the executable itself, and then combine your executable and the zip archive into a single contiguous file (similar to what zipapp does with it's helper script and app archive)
There are likely to still be rough edges when doing that, since this isn't a well tested configuration. When all else fails, find the part of the source code responsible for any error messages you're seeing, and try to work out if there's a setting you can tweak to avoid hitting that code path.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
-- Alberto García Illera GPG Public Key: https://goo.gl/twKUUv
Ohhhhhhhh I guess you mean this: https://github.com/anthony-tuininga/cx_Freeze/blob/master/source/bases/Commo... Right? On Mon, Jul 9, 2018 at 9:48 AM Alberto Garcia <agarciaillera@gmail.com> wrote:
Thank you for your response,
I was thinking on creating that zip file with the content of the Lib folder and having my c code to download it over the network and have it in memory. I guess that the zip file should have no compression at all right?
When you say that I need to use the cx_freeze approach what do you mean? Can you point me to where they do that?
And why changing sys.path again to the executable again? Which part of the executable?
I'll put my efforts in this.
Thank you
On Mon, Jul 9, 2018 at 7:16 AM Nick Coghlan <ncoghlan@gmail.com> wrote:
Hey there,
Yes, the part of having the pyd modules built in in library is already done. I followed the instructions in the README. What I would like to know now is how to embed the non frozen python (py) modules. Can you guys please
On 9 July 2018 at 03:10, Alberto Garcia <agarciaillera@gmail.com> wrote: point
me in the right direction.
The gist is to:
1. take the entire Lib directory and put it in a zip archive 2. use the approach demonstrated in cx_freeze to point sys.path in your static executable at that zip archive 3. adjust your C code to point sys.path back at the executable itself, and then combine your executable and the zip archive into a single contiguous file (similar to what zipapp does with it's helper script and app archive)
There are likely to still be rough edges when doing that, since this isn't a well tested configuration. When all else fails, find the part of the source code responsible for any error messages you're seeing, and try to work out if there's a setting you can tweak to avoid hitting that code path.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
-- Alberto García Illera
GPG Public Key: https://goo.gl/twKUUv
-- Alberto García Illera GPG Public Key: https://goo.gl/twKUUv
Does the zip need to reside in disk to be loaded. Or can it be loaded from memory? I don't want it to be loaded from disk but from Memory On Mon, Jul 9, 2018 at 9:59 AM Alberto Garcia <agarciaillera@gmail.com> wrote:
Ohhhhhhhh I guess you mean this:
https://github.com/anthony-tuininga/cx_Freeze/blob/master/source/bases/Commo...
Right?
On Mon, Jul 9, 2018 at 9:48 AM Alberto Garcia <agarciaillera@gmail.com> wrote:
Thank you for your response,
I was thinking on creating that zip file with the content of the Lib folder and having my c code to download it over the network and have it in memory. I guess that the zip file should have no compression at all right?
When you say that I need to use the cx_freeze approach what do you mean? Can you point me to where they do that?
And why changing sys.path again to the executable again? Which part of the executable?
I'll put my efforts in this.
Thank you
On Mon, Jul 9, 2018 at 7:16 AM Nick Coghlan <ncoghlan@gmail.com> wrote:
Hey there,
Yes, the part of having the pyd modules built in in library is already done. I followed the instructions in the README. What I would like to know now is how to embed the non frozen python (py) modules. Can you guys please
On 9 July 2018 at 03:10, Alberto Garcia <agarciaillera@gmail.com> wrote: point
me in the right direction.
The gist is to:
1. take the entire Lib directory and put it in a zip archive 2. use the approach demonstrated in cx_freeze to point sys.path in your static executable at that zip archive 3. adjust your C code to point sys.path back at the executable itself, and then combine your executable and the zip archive into a single contiguous file (similar to what zipapp does with it's helper script and app archive)
There are likely to still be rough edges when doing that, since this isn't a well tested configuration. When all else fails, find the part of the source code responsible for any error messages you're seeing, and try to work out if there's a setting you can tweak to avoid hitting that code path.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
-- Alberto García Illera
GPG Public Key: https://goo.gl/twKUUv
-- Alberto García Illera
GPG Public Key: https://goo.gl/twKUUv
-- Alberto García Illera GPG Public Key: https://goo.gl/twKUUv
I think you might find Gordon McMillian's installer interesting to look at. It has a lot f the tech that I think you are looking for. Works up to python 2.7. I ended up taking over when Gordon stopped maintaining it and kept it going up to python 2.7. In principle the same ideas could be made to work in python 3 I believe. https://sourceforge.net/projects/meinc-installer/ The zip file is appended to the end of the .EXE or unix ELF fie. The boot strap knows how to import form the ZIP at the end of the binary. It also has a way to split out the .SO/.DLL files from the ZIP and allow them to be loaded. Single EXE mode. There are docs that explain how it works in the sources. Barry
On 9 Jul 2018, at 19:40, Alberto Garcia <agarciaillera@gmail.com> wrote:
Does the zip need to reside in disk to be loaded. Or can it be loaded from memory? I don't want it to be loaded from disk but from Memory
On Mon, Jul 9, 2018 at 9:59 AM Alberto Garcia <agarciaillera@gmail.com <mailto:agarciaillera@gmail.com>> wrote: Ohhhhhhhh I guess you mean this: https://github.com/anthony-tuininga/cx_Freeze/blob/master/source/bases/Commo... <https://github.com/anthony-tuininga/cx_Freeze/blob/master/source/bases/Commo...> Right?
On Mon, Jul 9, 2018 at 9:48 AM Alberto Garcia <agarciaillera@gmail.com <mailto:agarciaillera@gmail.com>> wrote: Thank you for your response,
I was thinking on creating that zip file with the content of the Lib folder and having my c code to download it over the network and have it in memory. I guess that the zip file should have no compression at all right?
When you say that I need to use the cx_freeze approach what do you mean? Can you point me to where they do that?
And why changing sys.path again to the executable again? Which part of the executable?
I'll put my efforts in this.
Thank you
On Mon, Jul 9, 2018 at 7:16 AM Nick Coghlan <ncoghlan@gmail.com <mailto:ncoghlan@gmail.com>> wrote: On 9 July 2018 at 03:10, Alberto Garcia <agarciaillera@gmail.com <mailto:agarciaillera@gmail.com>> wrote:
Hey there,
Yes, the part of having the pyd modules built in in library is already done. I followed the instructions in the README. What I would like to know now is how to embed the non frozen python (py) modules. Can you guys please point me in the right direction.
The gist is to:
1. take the entire Lib directory and put it in a zip archive 2. use the approach demonstrated in cx_freeze to point sys.path in your static executable at that zip archive 3. adjust your C code to point sys.path back at the executable itself, and then combine your executable and the zip archive into a single contiguous file (similar to what zipapp does with it's helper script and app archive)
There are likely to still be rough edges when doing that, since this isn't a well tested configuration. When all else fails, find the part of the source code responsible for any error messages you're seeing, and try to work out if there's a setting you can tweak to avoid hitting that code path.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com <mailto:ncoghlan@gmail.com> | Brisbane, Australia -- Alberto García Illera
GPG Public Key: https://goo.gl/twKUUv <https://goo.gl/twKUUv>-- Alberto García Illera
GPG Public Key: https://goo.gl/twKUUv <https://goo.gl/twKUUv>
-- Alberto García Illera
GPG Public Key: https://goo.gl/twKUUv <https://goo.gl/twKUUv>_______________________________________________ Python-ideas mailing list Python-ideas@python.org <mailto:Python-ideas@python.org> https://mail.python.org/mailman/listinfo/python-ideas <https://mail.python.org/mailman/listinfo/python-ideas> Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/>
Hi, thank you for your response. I've downloaded the sources but I couldn't find any documentation. In addition I see that there is not a single C/C++ file. What I want to do is calling python from C. Am I missing something? Cheers On Mon, Jul 9, 2018 at 12:20 PM Barry Scott <barry@barrys-emacs.org> wrote:
I think you might find Gordon McMillian's installer interesting to look at. It has a lot f the tech that I think you are looking for.
Works up to python 2.7. I ended up taking over when Gordon stopped maintaining it and kept it going up to python 2.7. In principle the same ideas could be made to work in python 3 I believe.
https://sourceforge.net/projects/meinc-installer/
The zip file is appended to the end of the .EXE or unix ELF fie. The boot strap knows how to import form the ZIP at the end of the binary.
It also has a way to split out the .SO/.DLL files from the ZIP and allow them to be loaded. Single EXE mode.
There are docs that explain how it works in the sources.
Barry
On 9 Jul 2018, at 19:40, Alberto Garcia <agarciaillera@gmail.com> wrote:
Does the zip need to reside in disk to be loaded. Or can it be loaded from memory? I don't want it to be loaded from disk but from Memory
On Mon, Jul 9, 2018 at 9:59 AM Alberto Garcia <agarciaillera@gmail.com> wrote:
Ohhhhhhhh I guess you mean this:
https://github.com/anthony-tuininga/cx_Freeze/blob/master/source/bases/Commo...
Right?
On Mon, Jul 9, 2018 at 9:48 AM Alberto Garcia <agarciaillera@gmail.com> wrote:
Thank you for your response,
I was thinking on creating that zip file with the content of the Lib folder and having my c code to download it over the network and have it in memory. I guess that the zip file should have no compression at all right?
When you say that I need to use the cx_freeze approach what do you mean? Can you point me to where they do that?
And why changing sys.path again to the executable again? Which part of the executable?
I'll put my efforts in this.
Thank you
On Mon, Jul 9, 2018 at 7:16 AM Nick Coghlan <ncoghlan@gmail.com> wrote:
Hey there,
Yes, the part of having the pyd modules built in in library is already done. I followed the instructions in the README. What I would like to know now is how to embed the non frozen python (py) modules. Can you guys please
On 9 July 2018 at 03:10, Alberto Garcia <agarciaillera@gmail.com> wrote: point
me in the right direction.
The gist is to:
1. take the entire Lib directory and put it in a zip archive 2. use the approach demonstrated in cx_freeze to point sys.path in your static executable at that zip archive 3. adjust your C code to point sys.path back at the executable itself, and then combine your executable and the zip archive into a single contiguous file (similar to what zipapp does with it's helper script and app archive)
There are likely to still be rough edges when doing that, since this isn't a well tested configuration. When all else fails, find the part of the source code responsible for any error messages you're seeing, and try to work out if there's a setting you can tweak to avoid hitting that code path.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
-- Alberto García Illera
GPG Public Key: https://goo.gl/twKUUv
-- Alberto García Illera
GPG Public Key: https://goo.gl/twKUUv
-- Alberto García Illera
GPG Public Key: https://goo.gl/twKUUv _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- Alberto García Illera GPG Public Key: https://goo.gl/twKUUv
On 9 Jul 2018, at 21:11, Alberto Garcia <agarciaillera@gmail.com> wrote:
Hi,
thank you for your response. I've downloaded the sources but I couldn't find any documentation. In addition I see that there is not a single C/C++ file. What I want to do is calling python from C. Am I missing something?
You need to grab the sources with an svn checkout. I guess you grabbed the binary kit. https://sourceforge.net/p/meinc-installer/code/HEAD/tree/trunk/MEINC_Install... Docs are in here. https://sourceforge.net/p/meinc-installer/code/HEAD/tree/trunk/MEINC_Install... Barry
Cheers
On Mon, Jul 9, 2018 at 12:20 PM Barry Scott <barry@barrys-emacs.org> wrote: I think you might find Gordon McMillian's installer interesting to look at. It has a lot f the tech that I think you are looking for.
Works up to python 2.7. I ended up taking over when Gordon stopped maintaining it and kept it going up to python 2.7. In principle the same ideas could be made to work in python 3 I believe.
https://sourceforge.net/projects/meinc-installer/
The zip file is appended to the end of the .EXE or unix ELF fie. The boot strap knows how to import form the ZIP at the end of the binary.
It also has a way to split out the .SO/.DLL files from the ZIP and allow them to be loaded. Single EXE mode.
There are docs that explain how it works in the sources.
Barry
On 9 Jul 2018, at 19:40, Alberto Garcia <agarciaillera@gmail.com> wrote:
Does the zip need to reside in disk to be loaded. Or can it be loaded from memory? I don't want it to be loaded from disk but from Memory
On Mon, Jul 9, 2018 at 9:59 AM Alberto Garcia <agarciaillera@gmail.com> wrote: Ohhhhhhhh I guess you mean this: https://github.com/anthony-tuininga/cx_Freeze/blob/master/source/bases/Commo...
Right?
On Mon, Jul 9, 2018 at 9:48 AM Alberto Garcia <agarciaillera@gmail.com> wrote: Thank you for your response,
I was thinking on creating that zip file with the content of the Lib folder and having my c code to download it over the network and have it in memory. I guess that the zip file should have no compression at all right?
When you say that I need to use the cx_freeze approach what do you mean? Can you point me to where they do that?
And why changing sys.path again to the executable again? Which part of the executable?
I'll put my efforts in this.
Thank you
On Mon, Jul 9, 2018 at 7:16 AM Nick Coghlan <ncoghlan@gmail.com> wrote: On 9 July 2018 at 03:10, Alberto Garcia <agarciaillera@gmail.com> wrote: > Hey there, > > Yes, the part of having the pyd modules built in in library is already done. > I followed the instructions in the README. What I would like to know now is > how to embed the non frozen python (py) modules. Can you guys please point > me in the right direction.
The gist is to:
1. take the entire Lib directory and put it in a zip archive 2. use the approach demonstrated in cx_freeze to point sys.path in your static executable at that zip archive 3. adjust your C code to point sys.path back at the executable itself, and then combine your executable and the zip archive into a single contiguous file (similar to what zipapp does with it's helper script and app archive)
There are likely to still be rough edges when doing that, since this isn't a well tested configuration. When all else fails, find the part of the source code responsible for any error messages you're seeing, and try to work out if there's a setting you can tweak to avoid hitting that code path.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia -- Alberto García Illera
GPG Public Key: https://goo.gl/twKUUv -- Alberto García Illera
GPG Public Key: https://goo.gl/twKUUv
-- Alberto García Illera
GPG Public Key: https://goo.gl/twKUUv _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
-- Alberto García Illera
GPG Public Key: https://goo.gl/twKUUv
participants (5)
-
Alberto Garcia -
Barry -
Barry Scott
-
Nick Coghlan -
Paul Moore