Import Fails in setup.py On Android
After fixing a segfault issue (many thanks Ryan) I'm back to the same issue I was having with Python 2.7.8; the newly built python throws an undefined reference to dlopen when running setup.py...specifically when importing just-built extensions I've managed to narrow the problem down to the following line: importlib._bootstrap._SpecMethods(spec).load() Googling this brings up a few hits from bugs.python.org and not much else. I'm new to Python; any ideas what this does...or where I can read up on it for troubleshooting purposes? -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Update: While waiting for replies I made the change referenced here: https://bugs.python.org/review/5309/diff2/12811:12826/setup.py?context=3&column_width=80
specifically changing
importlib. _bootstrap._SpecMethods(spec).load()
to
importlib._bootstrap.load(spec)
I no longer get a terminating 'undefined reference to dlopen' error, but I do get 'importing extensions failed' errors for each extension...like this:
*** WARNING: importing extension "_pickle" failed with
After fixing a segfault issue (many thanks Ryan) I'm back to the same issue I was having with Python 2.7.8; the newly built python throws an undefined reference to dlopen when running setup.py...specifically when importing just-built extensions
I've managed to narrow the problem down to the following line:
importlib._bootstrap._SpecMethods(spec).load()
Googling this brings up a few hits from bugs.python.org and not much else. I'm new to Python; any ideas what this does...or where I can read up on it for troubleshooting purposes? -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
In reality, things just got broken even more. I don't know when that patch
was created, but it's now very out of date: importlib._bootstrap has no
load function. That's what the error you're getting is telling you. Since
it isn't getting to load anything, the issue seems "solved". Not really.
What's the full traceback for the undefined reference exception?
On Mon, Feb 2, 2015 at 2:04 PM, Cyd Haselton
Update: While waiting for replies I made the change referenced here: https://bugs.python.org/review/5309/diff2/12811:12826/setup.py?context=3&column_width=80
specifically changing importlib. _bootstrap._SpecMethods(spec).load() to importlib._bootstrap.load(spec)
I no longer get a terminating 'undefined reference to dlopen' error, but I do get 'importing extensions failed' errors for each extension...like this:
*** WARNING: importing extension "_pickle" failed with
: 'module' object has no attribute 'load' On February 2, 2015 1:36:29 PM CST, Cyd Haselton
wrote: After fixing a segfault issue (many thanks Ryan) I'm back to the same issue I was having with Python 2.7.8; the newly built python throws an undefined reference to dlopen when running setup.py...specifically when importing just-built extensions
I've managed to narrow the problem down to the following line:
importlib._bootstrap._SpecMethods(spec).load()
Googling this brings up a few hits from bugs.python.org and not much else. I'm new to Python; any ideas what this does...or where I can read up on it for troubleshooting purposes?
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
-- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." Personal reality distortion fields are immune to contradictory evidence. - srean Check out my website: http://kirbyfan64.github.io/
No traceback unfortunately...the fakechroot in the environment throws the error and setup.py fails.
I'll roll back that change...any idea where I could find info about the original method?
On February 2, 2015 3:17:54 PM CST, Ryan Gonzalez
In reality, things just got broken even more. I don't know when that patch was created, but it's now very out of date: importlib._bootstrap has no load function. That's what the error you're getting is telling you. Since it isn't getting to load anything, the issue seems "solved". Not really.
What's the full traceback for the undefined reference exception?
On Mon, Feb 2, 2015 at 2:04 PM, Cyd Haselton
wrote: Update: While waiting for replies I made the change referenced here:
https://bugs.python.org/review/5309/diff2/12811:12826/setup.py?context=3&column_width=80
specifically changing importlib. _bootstrap._SpecMethods(spec).load() to importlib._bootstrap.load(spec)
I no longer get a terminating 'undefined reference to dlopen' error,
do get 'importing extensions failed' errors for each extension...like
but I this:
*** WARNING: importing extension "_pickle" failed with
: 'module' object has no attribute 'load' On February 2, 2015 1:36:29 PM CST, Cyd Haselton
wrote:
After fixing a segfault issue (many thanks Ryan) I'm back to the
same
issue I was having with Python 2.7.8; the newly built python throws an undefined reference to dlopen when running setup.py...specifically when importing just-built extensions
I've managed to narrow the problem down to the following line:
importlib._bootstrap._SpecMethods(spec).load()
Googling this brings up a few hits from bugs.python.org and not much else. I'm new to Python; any ideas what this does...or where I can read up on it for troubleshooting purposes?
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
-- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." Personal reality distortion fields are immune to contradictory evidence. - srean Check out my website: http://kirbyfan64.github.io/
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Unfortunately, I have no idea. You might want to ask someone who's more
familiar with the Python source than I am.
What exactly appears? Does it say anything about the source of the error?
On Mon, Feb 2, 2015 at 3:24 PM, Cyd Haselton
No traceback unfortunately...the fakechroot in the environment throws the error and setup.py fails.
I'll roll back that change...any idea where I could find info about the original method?
On February 2, 2015 3:17:54 PM CST, Ryan Gonzalez
wrote: In reality, things just got broken even more. I don't know when that patch was created, but it's now very out of date: importlib._bootstrap has no load function. That's what the error you're getting is telling you. Since it isn't getting to load anything, the issue seems "solved". Not really.
What's the full traceback for the undefined reference exception?
On Mon, Feb 2, 2015 at 2:04 PM, Cyd Haselton
wrote: Update: While waiting for replies I made the change referenced here: https://bugs.python.org/review/5309/diff2/12811:12826/setup.py?context=3&column_width=80
specifically changing importlib. _bootstrap._SpecMethods(spec).load() to importlib._bootstrap.load(spec)
I no longer get a terminating 'undefined reference to dlopen' error, but I do get 'importing extensions failed' errors for each extension...like this:
*** WARNING: importing extension "_pickle" failed with
: 'module' object has no attribute 'load' On February 2, 2015 1:36:29 PM CST, Cyd Haselton
wrote: After fixing a segfault issue (many thanks Ryan) I'm back to the same issue I was having with Python 2.7.8; the newly built python throws an undefined reference to dlopen when running setup.py...specifically when importing just-built extensions
I've managed to narrow the problem down to the following line:
importlib._bootstrap._SpecMethods(spec).load()
Googling this brings up a few hits from bugs.python.org and not much else. I'm new to Python; any ideas what this does...or where I can read up on it for troubleshooting purposes?
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
-- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." Personal reality distortion fields are immune to contradictory evidence. - srean Check out my website: http://kirbyfan64.github.io/
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
-- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." Personal reality distortion fields are immune to contradictory evidence. - srean Check out my website: http://kirbyfan64.github.io/
It's not very helpful, but here it us if you want to look at it. I added the INFO/ALERT statements in setup.py for troubleshooting purposes.
INFO: removing statically built modules
INFO: Parsing Modules/Setup and Setup.local
INFO: Getting env var to pass to setup.py
INFO: Building remaining exts
ALERT: building extension "array" None
building 'array' extension
gcc --sysroot=/usr/gcc-4.9.2/sysroot -fPIC -Wno-unused-result -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -mandroid -mbionic -fno-builtin-sin -fno-builtin-cos -mandroid -mbionic -fno-builtin-sin -fno-builtin-cos -Werror=declaration-after-statement -I./Include -I. -IInclude -I/usr/gcc-4.9.2/sysroot/usr/include -I/bld/python/Python-3.4.2/Include -I/bld/python/Python-3.4.2 -c /bld/python/Python-3.4.2/Modules/arraymodule.c -o build/temp.linux-armv7l-3.4/bld/python/Python-3.4.2/Modules/arraymodule.o
gcc --sysroot=/usr/gcc-4.9.2/sysroot -shared -Wl,--dynamic-linker=/system/bin/linker -Wl,--sysroot=/usr/gcc-4.9.2/sysroot -Wl,--dynamic-linker=/system/bin/linker -Wl,--sysroot=/usr/gcc-4.9.2/sysroot -L/usr/local/lib -Wl,--dynamic-linker=/system/bin/linker -Wl,--sysroot=/usr/gcc-4.9.2/sysroot -L/usr/local/lib -mandroid -mbionic -fno-builtin-sin -fno-builtin-cos build/temp.linux-armv7l-3.4/bld/python/Python-3.4.2/Modules/arraymodule.o -L. -L/usr/local/lib -L/usr/gcc-4.9.2/sysroot/usr/lib -lc -ldl -lm -lpython3.4m -o build/lib.linux-armv7l-3.4/array.cpython-34m.so
INFO: building "array" complete None
ALERT: importing built module "array" None
fakechroot: dlopen: undefined symbol: dlopen
On February 2, 2015 3:32:38 PM CST, Ryan Gonzalez
Unfortunately, I have no idea. You might want to ask someone who's more familiar with the Python source than I am.
What exactly appears? Does it say anything about the source of the error?
On Mon, Feb 2, 2015 at 3:24 PM, Cyd Haselton
wrote: No traceback unfortunately...the fakechroot in the environment throws the error and setup.py fails.
I'll roll back that change...any idea where I could find info about the original method?
On February 2, 2015 3:17:54 PM CST, Ryan Gonzalez
wrote: In reality, things just got broken even more. I don't know when that patch was created, but it's now very out of date:
no load function. That's what the error you're getting is telling you. Since it isn't getting to load anything, the issue seems "solved". Not really.
What's the full traceback for the undefined reference exception?
On Mon, Feb 2, 2015 at 2:04 PM, Cyd Haselton
wrote: Update: While waiting for replies I made the change referenced here:
https://bugs.python.org/review/5309/diff2/12811:12826/setup.py?context=3&column_width=80
specifically changing importlib. _bootstrap._SpecMethods(spec).load() to importlib._bootstrap.load(spec)
I no longer get a terminating 'undefined reference to dlopen'
error, but
I do get 'importing extensions failed' errors for each extension...like this:
*** WARNING: importing extension "_pickle" failed with
: 'module' object has no attribute 'load' On February 2, 2015 1:36:29 PM CST, Cyd Haselton
wrote: After fixing a segfault issue (many thanks Ryan) I'm back to the
same
issue I was having with Python 2.7.8; the newly built python
importlib._bootstrap has throws an
undefined reference to dlopen when running setup.py...specifically when importing just-built extensions
I've managed to narrow the problem down to the following line:
importlib._bootstrap._SpecMethods(spec).load()
Googling this brings up a few hits from bugs.python.org and not much else. I'm new to Python; any ideas what this does...or where I can read up on it for troubleshooting purposes?
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
-- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." Personal reality distortion fields are immune to contradictory evidence. - srean Check out my website: http://kirbyfan64.github.io/
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
-- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." Personal reality distortion fields are immune to contradictory evidence. - srean Check out my website: http://kirbyfan64.github.io/
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
There’s now (as of a couple days ago) a python mobile-sig (https://mail.python.org/mailman/listinfo/mobile-sig). While that group is much smaller than python-dev and probably not as knowledgeable about the Cpython source base, that’s where you’re going to find folks who have a vested interest in getting Python working on mobile platforms like Android. (And who would be interested in hearing about your experiences, no matter whether they can help you or not.)
What you are doing (trying to run the entire C compiler toolchain on an Android machine, instead of using the cross-compilers that come in Google’s Android NDK (native development kit)) is way outside the mainstream. (I.e., you’re not going to find very many people trying to do this here or anywhere.)
Since you compiled all the libraries that you are linking against, there is some possibility (likelihood) that the problem is in one of those libraries, not anywhere in the CPython source. (Some other library was compiled without the necessary –dl flag.) The right way to track down this problem (no matter where it is) is to run under gdb and type “where” when the program crashes.
The reason I suspect that the problem is in one of the libraries you compiled before building Python, rather than a problem in the CPython sources or build scripts, is that I don’t have this problem when I cross-compile Python 3.4.2 on a Linux machine, then take the result and copy it over to the Android machine. I’ve posted instructions and patches for successfully performing this cross compilation (for Android KitKat running on an x86) here: https://github.com/wandering-logic/android_x86_python-3.4.
Best,
-Matt
From: Cyd Haselton [mailto:chaselton@gmail.com]
Sent: Monday, February 02, 2015 3:25 PM
To: Ryan Gonzalez
Cc: Python-Dev
Subject: Re: [Python-Dev] Import Fails in setup.py On Android
No traceback unfortunately...the fakechroot in the environment throws the error and setup.pyhttp://setup.py fails.
I'll roll back that change...any idea where I could find info about the original method?
On February 2, 2015 3:17:54 PM CST, Ryan Gonzalez
Reply in body of email below for clarity
On February 2, 2015 4:09:20 PM CST, "Frank, Matthew I"
There’s now (as of a couple days ago) a python mobile-sig (https://mail.python.org/mailman/listinfo/mobile-sig). While that group is much smaller than python-dev and probably not as knowledgeable about the Cpython source base, that’s where you’re going to find folks who have a vested interest in getting Python working on mobile platforms like Android. (And who would be interested in hearing about your experiences, no matter whether they can help you or not.)
Thanks for the notification. I'll definitely shoot them an email.
What you are doing (trying to run the entire C compiler toolchain on an Android machine, instead of using the cross-compilers that come in Google’s Android NDK (native development kit)) is way outside the mainstream. (I.e., you’re not going to find very many people trying to do this here or anywhere.)
There are several IDEs made for on-device compilation that include the GCC toolchain ported to the Android platform. http://www.spartacusrex.com/terminalide.htm http://kevinboone.net/kbox2_how_it_works.html I use the latter of the two (but contribute binaries to the former) because of its focus on emulating a Linux filesystem in which to run software.
Since you compiled all the libraries that you are linking against, there is some possibility (likelihood) that the problem is in one of those libraries, not anywhere in the CPython source. (Some other library was compiled without the necessary –dl flag.) The right way to track down this problem (no matter where it is) is to run under gdb and type “where” when the program crashes.
I did not compile all of the libraries that I'm linking against. The GCC port I am using makes use of a sysroot, which contains the necessary Android system libs and includes. The KBOX environment is built on a fakechroot that supports dynamic calls to Android's libc (and, I suspect, those functions that USUALLY reside in a normal libc...like dlopen) only. Provided a build includes the correct -ldl -lc links, it will work in KBOX.
The reason I suspect that the problem is in one of the libraries you compiled before building Python, rather than a problem in the CPython sources or build scripts, is that I don’t have this problem when I cross-compile Python 3.4.2 on a Linux machine, then take the result and copy it over to the Android machine. I’ve posted instructions and patches for successfully performing this cross compilation (for Android KitKat running on an x86) here: https://github.com/wandering-logic/android_x86_python-3.4.
See answer above. I'm working in an environment on the Android device. Hopefully this clarifies things. I'll re-post this over at python-sig.
Best, -Matt
From: Cyd Haselton [mailto:chaselton@gmail.com] Sent: Monday, February 02, 2015 3:25 PM To: Ryan Gonzalez Cc: Python-Dev Subject: Re: [Python-Dev] Import Fails in setup.py On Android
No traceback unfortunately...the fakechroot in the environment throws the error and setup.pyhttp://setup.py fails.
I'll roll back that change...any idea where I could find info about the original method? On February 2, 2015 3:17:54 PM CST, Ryan Gonzalez
mailto:rymg19@gmail.com> wrote: In reality, things just got broken even more. I don't know when that patch was created, but it's now very out of date: importlib._bootstrap has no load function. That's what the error you're getting is telling you. Since it isn't getting to load anything, the issue seems "solved". Not really. What's the full traceback for the undefined reference exception?
On Mon, Feb 2, 2015 at 2:04 PM, Cyd Haselton
mailto:chaselton@gmail.com> wrote: Update: While waiting for replies I made the change referenced here: https://bugs.python.org/review/5309/diff2/12811:12826/setup.py?context=3&column_width=80 specifically changing importlib. _bootstrap._SpecMethods(spec).load() to importlib._bootstrap.load(spec)
I no longer get a terminating 'undefined reference to dlopen' error, but I do get 'importing extensions failed' errors for each extension...like this:
*** WARNING: importing extension "_pickle" failed with
: 'module' object has no attribute 'load' On February 2, 2015 1:36:29 PM CST, Cyd Haselton
mailto:chaselton@gmail.com> wrote: After fixing a segfault issue (many thanks Ryan) I'm back to the same issue I was having with Python 2.7.8; the newly built python throws an undefined reference to dlopen when running setup.pyhttp://setup.py...specifically when importing just-built extensions I've managed to narrow the problem down to the following line:
importlib._bootstrap._SpecMethods(spec).load()
Googling this brings up a few hits from bugs.python.orghttp://bugs.python.org and not much else. I'm new to Python; any ideas what this does...or where I can read up on it for troubleshooting purposes?
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
_______________________________________________ Python-Dev mailing list Python-Dev@python.orgmailto:Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
-- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." Personal reality distortion fields are immune to contradictory evidence. - srean Check out my website: http://kirbyfan64.github.io/
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
On Mon, Feb 2, 2015 at 12:36 PM, Cyd Haselton
After fixing a segfault issue (many thanks Ryan) I'm back to the same issue I was having with Python 2.7.8; the newly built python throws an undefined reference to dlopen when running setup.py...specifically when importing just-built extensions
I've managed to narrow the problem down to the following line:
importlib._bootstrap._SpecMethods(spec).load()
That call is where modules are created (and executed) via the loader designated for the module by the import machinery. So a problem here may simply reflect a problem with the loader. Which module is being imported when you run into trouble? Is it a C-extension module? Also keep in mind that basically everything in importlib._bootstrap is frozen into your Python binary when you build Python (or should be). So if you are seeing any errors related to something missing or broken in importlib._bootstrap, it could be related to the build step where importlib gets frozen. Either way, more info (e.g. a traceback) would be great if you need more help. -eric
The array module is the current culprit.
Are there details about the freeze process at Python.org? That may help me figure out what and why my build breaks?
I'll try my best to get a traceback...I don't have a working gdb in this env yet...and if I do get one I'll add it to the email I send to python-sig
On February 3, 2015 12:10:54 AM CST, Eric Snow
On Mon, Feb 2, 2015 at 12:36 PM, Cyd Haselton
wrote: After fixing a segfault issue (many thanks Ryan) I'm back to the same issue I was having with Python 2.7.8; the newly built python throws an undefined reference to dlopen when running setup.py...specifically when importing just-built extensions
I've managed to narrow the problem down to the following line:
importlib._bootstrap._SpecMethods(spec).load()
That call is where modules are created (and executed) via the loader designated for the module by the import machinery. So a problem here may simply reflect a problem with the loader. Which module is being imported when you run into trouble? Is it a C-extension module?
Also keep in mind that basically everything in importlib._bootstrap is frozen into your Python binary when you build Python (or should be). So if you are seeing any errors related to something missing or broken in importlib._bootstrap, it could be related to the build step where importlib gets frozen.
Either way, more info (e.g. a traceback) would be great if you need more help.
-eric
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
While I'm waiting for a response from mobile-sig, I'm going through
the documentation (and something called "fossies") of importlib. I
hope that it's okay to ask some follow up questions...even though I've
redirected this post to mobile-sig.
If it is, my first question(s) is/are about the import machinery and
C-extension modules. Assuming that C-extension modules are ones that
include files that are named as *.c, how does the import machinery
handle specified shared libraries, such as -ldl? Additionally, how
does the import machinery handle the symbols from those libraries...if
it does at all?
On Tue, Feb 3, 2015 at 12:10 AM, Eric Snow
On Mon, Feb 2, 2015 at 12:36 PM, Cyd Haselton
wrote: After fixing a segfault issue (many thanks Ryan) I'm back to the same issue I was having with Python 2.7.8; the newly built python throws an undefined reference to dlopen when running setup.py...specifically when importing just-built extensions
I've managed to narrow the problem down to the following line:
importlib._bootstrap._SpecMethods(spec).load()
That call is where modules are created (and executed) via the loader designated for the module by the import machinery. So a problem here may simply reflect a problem with the loader. Which module is being imported when you run into trouble? Is it a C-extension module?
Also keep in mind that basically everything in importlib._bootstrap is frozen into your Python binary when you build Python (or should be). So if you are seeing any errors related to something missing or broken in importlib._bootstrap, it could be related to the build step where importlib gets frozen.
Either way, more info (e.g. a traceback) would be great if you need more help.
-eric
participants (4)
-
Cyd Haselton
-
Eric Snow
-
Frank, Matthew I
-
Ryan Gonzalez