[issue27126] Mac system sqlite3 not fork safe: Bundle a version?

[Evan Jones]

New submission from Evan Jones:

The system version of libsqlite3 that is included in Mac OS X is not fork safe. This means that if a process forks, and the child calls into it, it will crash with the stack trace below.

I've reproduced this with both Python 2.7.10 and Python 3.5.1 on Mac OS X 10.11.5. There are a number of reports about this issue on the Internet. The only way I can think to solve this problem is to bundle SQLite with the Python source code, and build an included version. This will avoid the problem, since only Apple's fork of SQLite uses the problematic libdispatch library.


Details:

* Apple ships a version of sqlite3 that uses their "Grand Central Dispatch" libdispatch library.

* Grand Central Dispatch is explicitly *not* thread safe. From their docs:

https://developer.apple.com/library/mac/documentation/Performance/Reference/GCD_libdispatch_Ref/index.html

"Be careful when mixing GCD with the fork system call. If a process makes GCD calls prior to calling fork, it is not safe to make additional GCD calls in the resulting child process until after a successful call to exec or related functions."

* Some System APIs also seem to call into this library. In my case: urllib/urllib2 access the system's proxy settings.


Related bugs:
* I believe this is the root cause of https://bugs.python.org/issue20353
* Celery also has a detailed bug report: https://github.com/celery/celery/issues/869
* Pure native code can cause this crash: http://ludovicrousseau.blogspot.com/2015/01/os-x-yosemite-bug-pcsc-functions-crash.html



Crash details from Mac OS X's system Python:

Application Specific Information:
crashed on child side of fork pre-exec

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libdispatch.dylib               0x00007fff92e8b162 _dispatch_barrier_async_f_slow + 356
1   libsqlite3.dylib                0x00007fff8a607656 sqlite3_initialize + 2950
2   libsqlite3.dylib                0x00007fff8a605b21 openDatabase + 65
3   _sqlite3.so                     0x0000000105bb9a55 pysqlite_connection_init + 509
4   org.python.python               0x000000010567bc24 0x10562f000 + 314404
5   org.python.python               0x0000000105639202 PyObject_Call + 99
6   _sqlite3.so                     0x0000000105bbdbf0 0x105bb8000 + 23536
7   org.python.python               0x00000001056b5a0b PyEval_EvalFrameEx + 13400
8   org.python.python               0x00000001056b8541 0x10562f000 + 562497
9   org.python.python               0x00000001056b530c PyEval_EvalFrameEx + 11609
10  org.python.python               0x00000001056b23c1 PyEval_EvalCodeEx + 1583
11  org.python.python               0x00000001056b1d8c PyEval_EvalCode + 54
12  org.python.python               0x00000001056d1a42 0x10562f000 + 666178
13  org.python.python               0x00000001056d1ae5 PyRun_FileExFlags + 133
14  org.python.python               0x00000001056d1634 PyRun_SimpleFileExFlags + 698
15  org.python.python               0x00000001056e3011 Py_Main + 3137
16  libdyld.dylib                   0x00007fff8eb225ad start + 1

----------
components: Macintosh
files: osx_python_crash.py
messages: 266396
nosy: evan.jones at bluecore.com, ned.deily, ronaldoussoren
priority: normal
severity: normal
status: open
title: Mac system sqlite3 not fork safe: Bundle a version?
versions: Python 2.7, Python 3.5
Added file: http://bugs.python.org/file43003/osx_python_crash.py

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue27126>
_______________________________________