[Python-Dev] Newly Built Python3 Binary Throws Segfault

[Cyd Haselton]

Very interesting. I got this error

Fatal Python error: Py_Initialize: Unable to get the locale encoding
NotImplementedError
Aborted
generate-posix-vars failed
make: *** [pybuilddir.txt] Error 1

...but it didn't (of course) segfault.  I'll pull gdb, get the results and send them.

On January 31, 2015 1:10:18 PM CST, Ryan <rymg19 at gmail.com> wrote:
>No; I was looking for all uses of _PyRaw_Strdup. Surprisingly, it's
>used only a few times.
>
>Cyd Haselton <chaselton at gmail.com> wrote:
>>Question:
>>When you said earlier that you found the problem by using grep...were
>>you looking for places where strdup called locale?
>>
>>On January 30, 2015 7:52:47 PM CST, Ryan Gonzalez <rymg19 at gmail.com>
>>wrote:
>>>Regardless, if you're looking to toy more with stuff like this, I'd
>>>highly
>>>recommend dual-booting with Ubuntu, which is what I'm doing now. (Now
>>I
>>>rarely ever boot into Windows!)
>>>
>>>On Fri, Jan 30, 2015 at 7:51 PM, Ryan Gonzalez <rymg19 at gmail.com>
>>>wrote:
>>>
>>>> Do you have just the SDK (which doesn't require Cygwin) or a rooted
>>>phone?
>>>> If so, I can upload instructions that don't use the NDK.
>>>>
>>>> On Fri, Jan 30, 2015 at 6:19 PM, Cyd Haselton <chaselton at gmail.com>
>>>wrote:
>>>>
>>>>> This is going to take some time...here's why:
>>>>>
>>>>> Downloading and installing the NDK/SDK won't be too hard...I have
>>to
>>>>> clear some space...but my primary machine is running Windows 7 and
>>>I'd
>>>>> rather swallow hot coals than install Cygwin.  I've got next to no
>>>>> experience with it, other than knowing that the NDK recommends
>>>against
>>>>> it.
>>>>>
>>>>> I've got a CentOS VM...but it's currently in tarball form on an
>>>>> external drive for space reasons...and it only has the NDK
>>>installed.
>>>>> If I am able to get it back up and running, there's still the task
>>>of
>>>>> getting  adb connected to my device. from the VM.
>>>>>
>>>>> Not saying it's impossible...just that it'll take time...and I'll
>>>>> probably have to tackle it tomorrow (earliest) or Sunday (latest).
>
>>>In
>>>>> the meantime I'll also check to see if there's anything that can
>a)
>>>>> run in an Android terminal and b) can take a stack trace; it would
>>>be
>>>>> far, far, far easier than either option above.
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Jan 30, 2015 at 4:19 PM, Ryan Gonzalez <rymg19 at gmail.com>
>>>wrote:
>>>>> > Do you have the Android SDK and NDK installed? If so...
>>>>> >
>>>>> > Using Google, I've created this series of steps, which may (or
>>may
>>>not)
>>>>> > work:
>>>>> >
>>>>> > 1. Make sure you have a copy of Python on your computer and make
>>>sure
>>>>> that
>>>>> > it's built with debug symbols.
>>>>> >
>>>>> > 2. Run the following commands from a shell with your phone
>>plugged
>>>in:
>>>>> >
>>>>> >   adb forward tcp:5039 tcp:5039
>>>>> >   adb shell /system/bin/gdbserver tcp:5039 <path to python
>>>executable>
>>>>> >   <path to ndk binary folder>/arm-linux-androideabi-gdb
>>>>> >
>>>>> > Now, GDB should have opened, so type the following commands:
>>>>> >
>>>>> >   set solib-search-path <directory holder libpython.so>
>>>>> >   file <path to local python executable>
>>>>> >   target remote :5055
>>>>> >   run
>>>>> >
>>>>> > Now, wait for the program to crash and type:
>>>>> >
>>>>> >   backtrace
>>>>> >
>>>>> > You should now see a complete backtrace in your terminal. To
>GDB,
>>>type:
>>>>> >
>>>>> >   quit
>>>>> >
>>>>> > *crosses fingers*
>>>>> >
>>>>> > On Fri, Jan 30, 2015 at 2:50 PM, Cyd Haselton
>>><chaselton at gmail.com>
>>>>> wrote:
>>>>> >>
>>>>> >> Unfortunately it is still reporting the same function :-/.
>>>>> >>
>>>>> >> On Fri, Jan 30, 2015 at 1:44 PM, Ryan Gonzalez
>><rymg19 at gmail.com>
>>>>> wrote:
>>>>> >> > Yes...
>>>>> >> >
>>>>> >> > Can you check if it's crashing in a different function now?
>>>>> >> >
>>>>> >> > On Fri, Jan 30, 2015 at 1:39 PM, Cyd Haselton
>>><chaselton at gmail.com>
>>>>> >> > wrote:
>>>>> >> >>
>>>>> >> >> Yes I did.  I did have to enter all the information in
>>>>> manually...I'm
>>>>> >> >> not familiar with automated patch application tools and even
>>>if I
>>>>> >> >> were, I'm pretty sure I wouldn't have them on my device.
>>>>> >> >>
>>>>> >> >> Just so that I'm absolutely sure I got everything
>>>correct...you
>>>>> wanted
>>>>> >> >> all of the lines in the patch commented out, correct? 
>>>Basically
>>>>> >> >> everything referencing oldloc?
>>>>> >> >>
>>>>> >> >> On Fri, Jan 30, 2015 at 1:00 PM, Ryan Gonzalez
>>><rymg19 at gmail.com>
>>>>> >> >> wrote:
>>>>> >> >> > Are you sure the patch was applied correctly? I was SO
>sure
>>>it
>>>>> would
>>>>> >> >> > work!
>>>>> >> >> >
>>>>> >> >> > FYI, you tried the patch I attached to the email message,
>>>right?
>>>>> >> >> >
>>>>> >> >> > On Fri, Jan 30, 2015 at 12:58 PM, Cyd Haselton <
>>>>> chaselton at gmail.com>
>>>>> >> >> > wrote:
>>>>> >> >> >>
>>>>> >> >> >> Update:  I did try the patch after getting it installed
>>>>> correctly,
>>>>> >> >> >> but
>>>>> >> >> >> I'm still getting a segfault on the newly built binary.
>>>>> >> >> >> Will post info this afternoon.
>>>>> >> >> >>
>>>>> >> >> >> On Fri, Jan 30, 2015 at 12:10 PM, Ryan Gonzalez <
>>>>> rymg19 at gmail.com>
>>>>> >> >> >> wrote:
>>>>> >> >> >> > No, it returns NULL if malloc gives it a raw pointer.
>It
>>>>> >> >> >> > unconditionally
>>>>> >> >> >> > checks the length of the (possibly null) string
>argument
>>>first.
>>>>> >> >> >> >
>>>>> >> >> >> > Please try the patch I attached in the last email. It
>>>*might*
>>>>> fix
>>>>> >> >> >> > the
>>>>> >> >> >> > issue.
>>>>> >> >> >> > Android has crappy locale handling.
>>>>> >> >> >> >
>>>>> >> >> >> > On Fri, Jan 30, 2015 at 12:09 PM, Cyd Haselton
>>>>> >> >> >> > <chaselton at gmail.com>
>>>>> >> >> >> > wrote:
>>>>> >> >> >> >>
>>>>> >> >> >> >>   Unless i'm reading something incorrectly, 
>>>_PyMem_RawStrdup
>>>>> is
>>>>> >> >> >> >> currently returning NULL when given a null pointer.
>>>>> >> >> >> >>
>>>>> >> >> >> >> From obmalloc.c
>>>>> >> >> >> >>
>>>>> >> >> >> >> _PyMem_RawStrdup(const char *str)
>>>>> >> >> >> >> {
>>>>> >> >> >> >>     size_t size;
>>>>> >> >> >> >> char *copy;
>>>>> >> >> >> >>     size = strlen(str) + 1;
>>>>> >> >> >> >>     copy = PyMem_RawMalloc(size);
>>>>> >> >> >> >>     if (copy == NULL)
>>>>> >> >> >> >>         return NULL;
>>>>> >> >> >> >> memcpy(copy, str, size);
>>>>> >> >> >> >> return copy;
>>>>> >> >> >> >>           }
>>>>> >> >> >> >>
>>>>> >> >> >> >> On Fri, Jan 30, 2015 at 11:56 AM, Ryan Gonzalez
>>>>> >> >> >> >> <rymg19 at gmail.com>
>>>>> >> >> >> >> wrote:
>>>>> >> >> >> >> > I seriously doubt the issue is in that file;
>>>>> _PyMem_RawStrdup
>>>>> >> >> >> >> > crashes
>>>>> >> >> >> >> > when
>>>>> >> >> >> >> > calling strlen. It's that whatever is calling it is
>>>likely
>>>>> >> >> >> >> > asking
>>>>> >> >> >> >> > it
>>>>> >> >> >> >> > to
>>>>> >> >> >> >> > duplicate a null pointer. Basically, it's probably
>>the
>>>>> caller's
>>>>> >> >> >> >> > fault.
>>>>> >> >> >> >> >
>>>>> >> >> >> >> > You could always try modifying _PyMem_RawStrdup to
>>>return
>>>>> NULL
>>>>> >> >> >> >> > when
>>>>> >> >> >> >> > given a
>>>>> >> >> >> >> > null pointer and see where it then segfaults.
>>>>> >> >> >> >> >
>>>>> >> >> >> >> > On Fri, Jan 30, 2015 at 11:53 AM, Cyd Haselton
>>>>> >> >> >> >> > <chaselton at gmail.com>
>>>>> >> >> >> >> > wrote:
>>>>> >> >> >> >> >>
>>>>> >> >> >> >> >> Alternatively, is there a hassle-free way to find
>>out
>>>what
>>>>> >> >> >> >> >> changed
>>>>> >> >> >> >> >> in
>>>>> >> >> >> >> >> obmalloc.c between 2.7.x and 3.4.x?
>>>>> >> >> >> >> >>
>>>>> >> >> >> >> >>
>>>>> >> >> >> >> >> On Fri, Jan 30, 2015 at 9:29 AM, Cyd Haselton
>>>>> >> >> >> >> >> <chaselton at gmail.com>
>>>>> >> >> >> >> >> wrote:
>>>>> >> >> >> >> >> > There's a related strdup patch for readline.c,
>>>mentioned
>>>>> >> >> >> >> >> > here:http://bugs.python.org/issue21390 and here
>>>>> >> >> >> >> >> >
>>>https://github.com/rave-engine/python3-android/issues/2.
>>>>> >> >> >> >> >> > There's a patch, but I'm not sure how to modify
>it
>>>for
>>>>> >> >> >> >> >> > obmalloc.c,
>>>>> >> >> >> >> >> > as
>>>>> >> >> >> >> >> > (I think) the functions all belong to
>>>Python...they're
>>>>> all
>>>>> >> >> >> >> >> > prefixed
>>>>> >> >> >> >> >> > with _PyXx
>>>>> >> >> >> >> >> >
>>>>> >> >> >> >> >> > On Fri, Jan 30, 2015 at 9:05 AM, Cyd Haselton
>>>>> >> >> >> >> >> > <chaselton at gmail.com>
>>>>> >> >> >> >> >> > wrote:
>>>>> >> >> >> >> >> >> Absolutely.  Good thing I have addr2line on
>>device
>>>>> >> >> >> >> >> >>
>>>>> >> >> >> >> >> >> /bld/python/Python-3.4.2 $ addr2line -C -f -e
>>>>> >> >> >> >> >> >> /lib/libpython3.4m.so.1.0
>>>>> >> >> >> >> >> >> 0008bbc8
>>>>> >> >> >> >> >> >> _PyMem_RawStrdup
>>>>> >> >> >> >> >> >> /bld/python/Python-3.4.2/Objects/obmalloc.c:323
>>>>> >> >> >> >> >> >> /bld/python/Python-3.4.2 $
>>>>> >> >> >> >> >> >>
>>>>> >> >> >> >> >> >>
>>>>> >> >> >> >> >> >>
>>>>> >> >> >> >> >> >> On Thu, Jan 29, 2015 at 8:26 PM, Ryan
>>><rymg19 at gmail.com
>>>>> >
>>>>> >> >> >> >> >> >> wrote:
>>>>> >> >> >> >> >> >>> Could you try the steps at
>>>>> >> >> >> >> >> >>> http://stackoverflow.com/a/11369475/2097780?
>>They
>>>>> >> >> >> >> >> >>> allow you to get a better idea of where libc is
>>>>> crashing.
>>>>> >> >> >> >> >> >>>
>>>>> >> >> >> >> >> >>> Cyd Haselton <chaselton at gmail.com> wrote:
>>>>> >> >> >> >> >> >>>>
>>>>> >> >> >> >> >> >>>> Managed to get this out of logcat:
>>>>> >> >> >> >> >> >>>> F(11914) Fatal signal 11 (SIGSEGV) at
>>0x00000000
>>>>> >> >> >> >> >> >>>> (code=1),
>>>>> >> >> >> >> >> >>>> thread
>>>>> >> >> >> >> >> >>>> 11914 (python)  (libc)
>>>>> >> >> >> >> >> >>>>
>>>>> >> >> >> >> >> >>>> [ 01-29 19:30:55.855 23373:23373 F/libc     ]
>>>>> >> >> >> >> >> >>>> Fatal signal 11 (SIGSEGV) at 0x00000000
>>>(code=1),
>>>>> thread
>>>>> >> >> >> >> >> >>>> 23373
>>>>> >> >> >> >> >> >>>> (python)
>>>>> >> >> >> >> >> >>>>
>>>>> >> >> >> >> >> >>>> Less detail than strace but it seems to be
>that
>>>>> python is
>>>>> >> >> >> >> >> >>>> segfaulting
>>>>> >> >> >> >> >> >>>> libc...
>>>>> >> >> >> >> >> >>>>
>>>>> >> >> >> >> >> >>>> On Wed, Jan 28, 2015 at 11:23 AM, Ryan
>Gonzalez
>>>>> >> >> >> >> >> >>>> <rymg19 at gmail.com>
>>>>> >> >> >> >> >> >>>> wrote:
>>>>> >> >> >> >> >> >>>>>
>>>>> >> >> >> >> >> >>>>>  On Wed, Jan 28, 2015 at 10:43 AM, Guido van
>>>Rossum
>>>>> >> >> >> >> >> >>>>> <guido at python.org>
>>>>> >> >> >> >> >> >>>>> wrote:
>>>>> >> >> >> >> >> >>>>>>
>>>>> >> >> >> >> >> >>>>>>
>>>>> >> >> >> >> >> >>>>>>  What I see in the strace:
>>>>> >> >> >> >> >> >>>>>>
>>>>> >> >> >> >> >> >>>>>>  ... load libpython3.4m.so.1.0
>>>>> >> >> >> >> >> >>>>>>  ... load libm
>>>>> >> >> >> >> >> >>>>>>  ... open /dev/__properti
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150131/d49be8ae/attachment-0001.html>