Wrong OSX platform in sysconfig.py causing installation problems
Hello Everyone, my first post. I would also like to subscribe but haven't seen how yet. So I've an issue with how the 3.5.1 OSX package installs on OSX. The problem was first made apparent when trying to create a virtual environment and `pip install ZODB`, which failed as clang and the linker could not find either the include or lib directories for a dependency. By adding a -v option to the compiler options I was able to see in the compiler flags the following: -isysroot /Developer/SDKs/MacOSX10.6.sdk Which was not present in the compiler flags of other people who I was talking with. The isysroot flag changes the location of includes and libs from the system default, which would be why they can't be found. After grepping around a bit I found this file: /Library/Frameworks/Python.framework/Versions/3.5 /lib/python3.5/_sysconfigdata.py Within which are many mentions of this flag, as so: 'CONFIGURE_CFLAGS': '-arch i386 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.6.sdk -g', 'CONFIGURE_CFLAGS_NODIST': '-Werror=declaration-after -statement', 'CONFIGURE_CPPFLAGS': '-isysroot /Developer/SDKs/MacOSX10.6.sdk', 'CONFIGURE_LDFLAGS': '-arch i386 -arch x86_64 -isysroot /Developer/SDKs/MacOSX10.6.sdk -g', 'CONFIG_ARGS': "'-C' '--enable-framework' '--enable- universalsdk=/Developer/SDKs/MacOSX10.6.sdk' '--with-universal-archs=intel' '--with-computed-gotos' '--without-ensurepip' 'LDFLAGS=-g' 'CFLAGS=-g' 'CC=gcc-4.2'", among others. By running Python 3.5's sysconfig.py as a module, via `python -m sysconfig`, I see the following at the beginning of the output: Platform: "macosx-10.6-intel" Python version: "3.5" Current installation scheme: "posix_prefix" And when I simply go into python interactive mode, import sysconfig, and type sysconfig.get_platform() it returns macosx-10.6-intel. I believe somehow the platform being reported is incorrect, but looking at my environment, I do not have any kind of compiler flags or anything else like that set in my bash init files. Can someone who has a working OSX 3.5 installation check their reported platform to see what it is and if anyone has an idea as to how this information is getting into the installation and where I might change it please let me know. Sincerely, Joseph josephnunn@gmail.com
On May 6, 2016, at 05:42, Joseph Nunn <josephnunn@gmail.com> wrote:
Hello Everyone, my first post. I would also like to subscribe but haven't seen how yet.
So I've an issue with how the 3.5.1 OSX package installs on OSX. The problem was first made apparent when trying to create a virtual environment and `pip install ZODB`, which failed as clang and the linker could not find either the include or lib directories for a dependency. By adding a -v option to the compiler options I was able to see in the compiler flags the following:
-isysroot /Developer/SDKs/MacOSX10.6.sdk
Which was not present in the compiler flags of other people who I was talking with. [...]
Hi Joseph! Welcome! Note this list is for the development of Python itself, not for usage or installation issues. The Python Developer's Guide (https://docs.python.org/devguide/) including its FAQ gives information about mailing lists and other support forums to get help with general Python issues. That said, without more information, it would be difficult to know exactly what problem you are running into but, since you mention missing headers, my guess is that you need to install Apple's Command Line Tools as described here: https://docs.python.org/devguide/setup.html#build-dependencies Installing Xcode by itself is not sufficient. The sysroot and flag values you note in _sysconfigdata.py are normal and there for compatibility with installing on older versions of OS X; they should not cause a problem on newer releases. For what it's worth, I was able to install ZODB with the current python.org 3.5.1 on OS X 10.11.4 using either venv or virtualenv. Good luck! --Ned P.S. Mailing list info is available here: https://mail.python.org. The bug tracker is here: https://bugs.python.org -- Ned Deily nad@python.org -- []
Then I have found the right list to post on, for my problem is a bug in the Python 3.5.1 distribution. Albeit one which will only manifest in certain situations. The presence of numerous `-isysroot /Developer/SDKs/MacOSX10.6.sdk` directives in the following file /Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/_sysconfigdata.py Will cause errors for people who have kept around the old /Developer directory tree, and old SDKs like 10.6, for building backwards compatible old code. Newer versions of the SDKs are now kept inside the Xcode application itself. And then they will only have a problem when trying to install a package with C extensions, which in my case was Persistence, a dependency of ZODB. And even more so, ONLY when trying to pip install said package into a virtual environment created by 3.5’s pyvenv command, if installing the package into the system wide Python there will be no error. Such is this corner-case hell I have found myself, until I decided to dump my older /Developer directory tree and SDKs, as those were for a project long defunct. You see the -isysroot flag has the unusual behavior where instead of reporting an error when it cannot find the newly specified system root it simply has no effect. So if you don’t have the SDK specified in the location specified then the flag does nothing, as I assume is the case for 99% of people. I understand the need to make Python 3.5 backwards compatible, but doing it this way would appear to have the side effect of breaking the ability to install packages using C extensions in virtual environments, or at least it did for me. Joseph Nunn josephnunn@gmail.com joseph.nunn@uci.edu
On May 06 , 2016, at 4:20 AM, Ned Deily <nad@python.org> wrote:
On May 6, 2016, at 05:42, Joseph Nunn <josephnunn@gmail.com> wrote:
Hello Everyone, my first post. I would also like to subscribe but haven't seen how yet.
So I've an issue with how the 3.5.1 OSX package installs on OSX. The problem was first made apparent when trying to create a virtual environment and `pip install ZODB`, which failed as clang and the linker could not find either the include or lib directories for a dependency. By adding a -v option to the compiler options I was able to see in the compiler flags the following:
-isysroot /Developer/SDKs/MacOSX10.6.sdk
Which was not present in the compiler flags of other people who I was talking with. [...]
Hi Joseph!
Welcome! Note this list is for the development of Python itself, not for usage or installation issues. The Python Developer's Guide (https://docs.python.org/devguide/) including its FAQ gives information about mailing lists and other support forums to get help with general Python issues. That said, without more information, it would be difficult to know exactly what problem you are running into but, since you mention missing headers, my guess is that you need to install Apple's Command Line Tools as described here:
https://docs.python.org/devguide/setup.html#build-dependencies
Installing Xcode by itself is not sufficient. The sysroot and flag values you note in _sysconfigdata.py are normal and there for compatibility with installing on older versions of OS X; they should not cause a problem on newer releases. For what it's worth, I was able to install ZODB with the current python.org 3.5.1 on OS X 10.11.4 using either venv or virtualenv.
Good luck!
--Ned
P.S. Mailing list info is available here: https://mail.python.org. The bug tracker is here: https://bugs.python.org
-- Ned Deily nad@python.org -- []
On Fri, May 6, 2016 at 10:43 AM, Joseph Lee Nunn III <josephnunn@gmail.com> wrote:
Then I have found the right list to post on, for my problem is a bug in the Python 3.5.1 distribution. Albeit one which will only manifest in certain situations.
The presence of numerous `-isysroot /Developer/SDKs/MacOSX10.6.sdk` directives in the following file
/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/_sysconfigdata.py
As I understand it, that is all there so that when extensions are built, they are linked against that SDK -- so you can build extensions that can then be run on any system 10.6 and greater.
Will cause errors for people who have kept around the old /Developer directory tree, and old SDKs like 10.6, for building backwards compatible old code.
what are the errors -- wouldn't the old SDK be found, and used, and that's exactly what's intended?
Newer versions of the SDKs are now kept inside the Xcode application itself. And then they will only have a problem when trying to install a package with C extensions,
well, sure. of course this is an issue for building C extensions.
which in my case was Persistence, a dependency of ZODB. And even more so, ONLY when trying to pip install said package into a virtual environment created by 3.5’s pyvenv command, if installing the package into the system wide Python there will be no error.
hmm, As I understand it the building process should be exactly the same whether you are on a virtualenv or not. but if so -- maybe the "solution" is to build a wheel outside virtualenv, and then install that wheel in the virtualenv. But I agree, it should work, of course.
You see the -isysroot flag has the unusual behavior where instead of reporting an error when it cannot find the newly specified system root it simply has no effect. So if you don’t have the SDK specified in the location specified then the flag does nothing, as I assume is the case for 99% of people.
Indeed, that's been working fine for me for a while. However it's this behavior that is actually the bug -- extensions are supposed to be build against the 10.6 SDK -- and that's not what's happening in this case
I understand the need to make Python 3.5 backwards compatible, but doing it this way would appear to have the side effect of breaking the ability to install packages using C extensions in virtual environments, or at least it did for me.
It does sound like you've hit a bug, but I actually think you've misidentified the bug -- I'm trying to understand why it's not working in a virtualenv -- THAT's the bug. And I have no idea what to do about it -- thanks Apple for moving things -- that's a great way to keep backward compatibility! Ned: I'm trying to explore this a bit now (on a 10.9 system), so I followed the instructions in the Python developers guide, and did: xcode-select --install And was a bit surprised that It did a substantial download -- I thought I already had the XCode command line tools -- but whatever. After running that, I found I had the 10.9 and 10.10 SDKs (I think that installed the 10.10 one, I previously had only 10.9). And they are buried deep in XCode.app .... OK, but how would I go about getting the 10.6 SDK??? This SO post: http://stackoverflow.com/questions/11989007/where-to-get-macos-sdk-10-6-for-... indicates that it's a serious pain in the $%^% -- thank Apple! But, as far as I can tell, if you install an old XCode, you will get the older SDKs, and installed into the old location. (this is why I kept a 10.6 system around for a long time -- to build python extensions...) But what should we do? IF you are building extensions only for yourself, then I think whatever the heck SDK you have will work fine -- but how to build for distribution? .... Trying to decide if I want to go through the hassle of installing an old XCode to see if i can replicate the bug the OP is reporting.... -CHB
https://docs.python.org/devguide/setup.html#build-dependencies
Installing Xcode by itself is not sufficient. The sysroot and flag
values you note in _sysconfigdata.py are normal and there for compatibility with installing on older versions of OS X; they should not cause a problem on newer releases. For what it's worth, I was able to install ZODB with the current python.org 3.5.1 on OS X 10.11.4 using either venv or virtualenv.
Good luck!
--Ned
P.S. Mailing list info is available here: https://mail.python.org. The
bug tracker is here: https://bugs.python.org
-- Ned Deily nad@python.org -- []
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Sat, May 7, 2016 at 10:59 PM, Chris Barker <chris.barker@noaa.gov> wrote:
.... Trying to decide if I want to go through the hassle of installing an old XCode to see if i can replicate the bug the OP is reporting....
OK, I found this gitHub repo with all the old SDKs: https://github.com/phracker/MacOSX-SDKs I put a copy of the 10.6 SDk in it's old location: /Developer/SDKs/MacOSX10.6.sdk/ /before I did that, I did a pip install of zodb in a virtualenv, and all seemed to go fine. THe extensions rae linked against : $ otool -L cPersistence.cpython-35m-darwin.so cPersistence.cpython-35m-darwin.so: /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1) where that comes from -- who knows?? then I tried again, after putting the sdk in place: [note: serious pain the $%^# to get pip to stop using it's cache! ] and everything seemed to build fine, and at least one simple test of zodb works fine. And this is how the extensions are linked: $ otool -L cPersistence.cpython-35m-darwin.so cPersistence.cpython-35m-darwin.so: /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.11) which is the same as above, but the "current version" is much older -- so does that mean it used the 10.6 SDK???? Anyway, I can't reproduce the OP's bug..... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
Thank you for looking into this for me, but since I fixed my problem by simply deleting the old SDKs I’ve moved on. However let me clarify exactly what was going on one more time, just in case you guys decide to pursue further. I had the old 10.6 SDK in /Developer/SDKs, it could have been ‘special’ in that I did alot of weird things for a former project and I just don’t recall. For the sake of explanation though lets say it wasn’t, what then? I installed OSX Python 3.5.1, created a virtual environment using the pyvenv command from that newly installed Python 3.5.1, and sourced its bin/activate. Then when running pip install ZODB, the compiler would be passed the -isysroot flag for the 10.6 SDK, which would change its include paths and it would find no includes or libs (if you sym link the includes into the include directory of the virtual env you will get farther and see that the linker does not find libs either). By suppling adding -v to CFLAGS you can have the compiler output the include paths and compiler call when running pip install. That is how I noticed the alternate system root supplied by the isysroot flag. Grepping around found me the _sysconfigdata.py file where that stuff was specified, it took a bit to realize that file came with the distribution and wasn’t generated on my system. When not in a virtual environment, ZODB would install just fine into the system python (2.7). Not sure why that is, or what would make the virtual environment different, but there you go. Joseph Nunn josephnunn@gmail.com joseph.nunn@uci.edu
On May 07 , 2016, at 11:52 PM, Chris Barker <chris.barker@noaa.gov> wrote:
On Sat, May 7, 2016 at 10:59 PM, Chris Barker <chris.barker@noaa.gov <mailto:chris.barker@noaa.gov>> wrote:
.... Trying to decide if I want to go through the hassle of installing an old XCode to see if i can replicate the bug the OP is reporting....
OK, I found this gitHub repo with all the old SDKs:
https://github.com/phracker/MacOSX-SDKs <https://github.com/phracker/MacOSX-SDKs>
I put a copy of the 10.6 SDk in it's old location:
/Developer/SDKs/MacOSX10.6.sdk/
/before I did that, I did a pip install of zodb in a virtualenv, and all seemed to go fine. THe extensions rae linked against :
$ otool -L cPersistence.cpython-35m-darwin.so <http://cpersistence.cpython-35m-darwin.so/> cPersistence.cpython-35m-darwin.so <http://cpersistence.cpython-35m-darwin.so/>: /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
where that comes from -- who knows??
then I tried again, after putting the sdk in place:
[note: serious pain the $%^# to get pip to stop using it's cache! ]
and everything seemed to build fine, and at least one simple test of zodb works fine. And this is how the extensions are linked:
$ otool -L cPersistence.cpython-35m-darwin.so <http://cpersistence.cpython-35m-darwin.so/> cPersistence.cpython-35m-darwin.so <http://cpersistence.cpython-35m-darwin.so/>: /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.11)
which is the same as above, but the "current version" is much older -- so does that mean it used the 10.6 SDK????
Anyway, I can't reproduce the OP's bug.....
-CHB
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov <mailto:Chris.Barker@noaa.gov>
Thanks Joseph, I say this isn't worth pursuing until someone else reports the issue, as I couldn't replicate it with a "clean" system. I had the old 10.6 SDK in /Developer/SDKs, it could have been ‘special’ in
that I did alot of weird things for a former project and I just don’t recall.
I suspect your 10.6 SDK was cnfigured oddly, or ???? For the sake of explanation though lets say it wasn’t, what then?
I installed OSX Python 3.5.1, created a virtual environment using the pyvenv command from that newly installed Python 3.5.1, and sourced its bin/activate. Then when running pip install ZODB, the compiler would be passed the -isysroot flag for the 10.6 SDK, which would change its include paths and it would find no includes or libs
this is the odd part -- shouldn't the new sysroot (i.e. the SDK) have all the includes and libs needed? Not sure why that is, or what would make the virtual environment different,
but there you go.
yeah, me neither .... Anyway, unless someone else sees this, I think we're done. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On May 09 , 2016, at 11:03 AM, Chris Barker <chris.barker@noaa.gov> wrote:
I installed OSX Python 3.5.1, created a virtual environment using the pyvenv command from that newly installed Python 3.5.1, and sourced its bin/activate. Then when running pip install ZODB, the compiler would be passed the -isysroot flag for the 10.6 SDK, which would change its include paths and it would find no includes or libs
this is the odd part -- shouldn't the new sysroot (i.e. the SDK) have all the includes and libs needed?
That is the weird part, if installing into the system python the new sysroot worked just fine. It was only when trying to install into a virtual environment that it failed. What the difference is I just don’t know. Thanks again for you efforts! Joseph
participants (4)
-
Chris Barker -
Joseph Lee Nunn III -
Joseph Nunn
-
Ned Deily