Mailman 3 February 2016 - Distutils-SIG

PEP: Build system abstraction for pip/conda etc
by Donald Stufft 12 Feb '16

12 Feb '16

As many know there has been an effort to get a generalized interface that a build system can implement so that we can break the hard dependency on setuptools that Python packaging currently has. After a lot of deliberation and threads back and forth (as well as video meetings!) I think that the version of the PEP that is currently on the PR in https://github.com/pypa/interoperability-peps/pull/54 looks like it’s generally the right thing to move forward with. I made a few little comments but overall I think it’s there and we’re ready to move forward on trying to get a reference implementation done that can validate the decisions made in that PEP (and then, hopefully finalize the PEP and merge those implementations). So many thanks to everyone involved in hammering this out thus far :) I’m nervous but excited about the possibility of making setuptools just another build system. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

14 55

Updates to PEP-513
by Nathaniel Smith 09 Feb '16

09 Feb '16

Hi all, Based on recent discussions, I'm proposing the following update to PEP 513: https://github.com/pypa/manylinux/pull/31/files The changes to the text are: * Remove the UCS-4 requirement in favor of pip/wheel's new SOABI support. * Mandate that manylinux1 extensions not link against libpythonX.Y.so.1, for the complicated compatibility-related reasons that were discussed on list. * Update link for the docker images to their new PyPA home. Comments welcome, of course. I haven't sent this to the PEP editors yet, because I'm not sure if there's any particular ceremony needed for changing the semantics of a PEP that's already been pronounced upon? -n -- Nathaniel J. Smith -- https://vorpus.org

2 1

Does anyone understand what's going on with libpython on Linux?
by Nathaniel Smith 08 Feb '16

08 Feb '16

So we found another variation between how different distros build CPython [1], and I'm very confused. Fedora (for example) turns out to work the way I naively expected: taking py27 as our example, they have: - libpython2.7.so.1.0 contains the actual python runtime - /usr/bin/python2.7 is a tiny (~7 KiB) executable that links to libpython2.7.so.1 to do the actual work; the main python package depends on the libpython package - python extension module packages depend on the libpython package, and contain extension modules linked against libpython2.7.so.1 - python extension modules compiled locally get linked against libpython2.7.so.1 by default Debian/Ubuntu do things differently: - libpython2.7.so.1.0 exists and contains the full python runtime, but is not installed by default - /usr/bin/python2.7 *also* contains a *second* copy of the full python runtime; there is no dependency relationship between these, and you don't even get libpython2.7.so.1.0 installed unless you explicitly request it or it gets pulled in through some other dependency - most python extension module packages do *not* depend on the libpython2.7 package, and contain extension modules that are *not* linked against libpython2.7.so.1.0 (but there are exceptions!) - python extension modules compiled locally do *not* get linked against libpython2.7.so.1 by default. The only things that seem to link against libpython2.7.so.1.0 in debian are: a) other packages that embed python (e.g. gnucash, paraview, perf, ...) b) some minority of python packages (e.g. the PySide/QtOpenGL.so module is one that I found that directly links to libpython2.7.so.1.0) I guess that the reason this works is that according to ELF linking rules, the symbols defined in the main executable, or in the transitive closure of the libraries that the main executable is linked to via DT_NEEDED entries, are all injected into the global scope of any dlopen'ed libraries. Uh, let me try saying that again. When you dlopen() a library -- like, for example, a python extension module -- then the extension automatically gets access to any symbols that are exported from either (a) the main executable itself, or (b) any of the libraries that are listed if you run 'ldd <the main executable>'. It also gets access to any symbols that are exported by itself, or any of the libraries listed if you run 'ldd <the dlopen'ed library>'. OTOH it does *not* get access to any symbols exported by other libraries that get dlopen'ed -- each dlopen specifically creates its own "scope". So the reason this works is that Debian's /usr/bin/python2.7 itself exports all the standard Python C ABI symbols, so any extension module that it loads automatically get access to the CPython ABI, even if they don't explicitly link to it. And programs like gnucash are linked directly to libpython2.7.so.1, so they also end up exporting the CPython ABI to any libraries that they dlopen. But, it seems to me that there are two problems with the Debian/Ubuntu way of doing things: 1) it's rather wasteful of space, since there are two complete independent copies of the whole CPython runtime (one inside /usr/bin/python2.7, the other inside libpython2.7.so.1). 2) if I ever embed cpython by doing dlopen("libpython2.7.so.1"), or dlopen("some_plugin_library_linked_to_libpython.so"), then the embedded cpython will not be able to load python extensions that are compiled in the Debian-style (but will be able to load python extensions compiled in the Fedora-style), because the dlopen() the loaded the python runtime and the dlopen() that loads the extension module create two different scopes that can't see each other's symbols. [I'm pretty sure this is right, but linking is arcane and probably I should write some tests to double check.] I guess (2) might be why some of Debian's extension modules do link to libpython2.7.so.1 directly? Or maybe that's just a bug? Is there any positive reason in favor of the Debian style approach? Clearly someone put some work into setting things up this way, so there must be some motivation, but I'm not sure what it is? The immediate problem for us is that if a manylinux1 wheel links to libpythonX.Y.so (Fedora-style), and then it gets run on a Debian system that doesn't have libpythonX.Y.so installed, it will crash with: ImportError: libpython2.7.so.1.0: cannot open shared object file: No such file or directory Maybe this is okay and the solution is to tell people that they need to 'apt install libpython2.7'. In a sense this isn't even a regression, because every system that is capable of installing a binary extension from an sdist has python2.7-dev installed, which depends on libpython2.7 --> therefore every system that used to be able to do 'pip install somebinary' with sdists will still be able to do it with manylinux1 builds. The alternative is to declare that manylinux1 extensions should not link to libpython. This should I believe work fine on both Debian-style and Fedora-style installations -- though the PySide example, and the theoretical issue with embedding python through dlopen, both give me some pause. Two more questions: - What are Debian/Ubuntu doing in distutils so that extensions don't link to libpython by default? If we do go with the option of saying that manylinux extensions shouldn't link to libpython, then that's something auditwheel *can* fix up, but it'd be even nicer if we could set up the docker image to get it right in the first place. - Can/should Debian/Ubuntu switch to the Fedora model? Obviously it would take quite some time before a generic platform like manylinux could assume that this had happened, but it does seem better to me...? And if it's going to happen at all it might be nice to get the switch into 16.04 LTS? Of course that's probably ambitious, even if I'm not missing some reason why the Debian/Ubuntu model is actually advantageous. -n [1] https://github.com/pypa/manylinux/issues/30 -- Nathaniel J. Smith -- https://vorpus.org

6 9

SOABI for Unicode ABI on 2.x (was: wheel 0.27.0 released)
by Nathaniel Smith 08 Feb '16

08 Feb '16

On Feb 5, 2016 8:47 AM, "Nate Coraor" <nate(a)bx.psu.edu> wrote: > [...] > - Add SOABI tags to platform-specific wheels built for Python 2.X (Pull Request > #55, Issue #63, Issue #101) I can't quite untangle all the documents linked from this PR, so let me ask here :-). Does this mean that python 2.x extension wheels now can and should declare whether they're assuming the 16- or 32-bit Unicode ABI inside the abi field? And if so, should PEP 513 be updated to allow for both options to be used with manylinux1? (Right not manylinux1 just implies/requires a UCS4 build, for older pythons where this matters.) -n

6 10

wheel 0.28.0 released
by Nate Coraor 06 Feb '16

06 Feb '16

Hi all, There was a bug introduced in 0.27.0 where scripts in the wheel archive were created with the wrong permissions. This has been fixed and released in 0.28.0. --nate

2 2

Status update on the NumPy & SciPy vs SSE problem?
by Nick Coghlan 06 Feb '16

06 Feb '16

While the manylinux PEP brings Linux up to comparable standing with Windows and Mac OS X in terms of distributing wheel files through PyPI, that does mean it still suffers from the same problem Windows does in relation to NumPy and SciPy wheels: no standardisation of the SSE capabilities of the machines. I figured that was independent of the manylinux PEP (since it affects Windows as well), but I'm also curious as to the current status (I found a couple of apparently relevant threads on the NumPy list, but figured it made more sense to just ask for an update rather than trusting my Google-fu) Cheers, Nick. P.S. I'm assuming the existing ability to publish NumPy & SciPy wheels for Mac OS X is based on Apple's tighter control of their hardware platform. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

5 7

How to declare optional dependencies?
by Brett Cannon 06 Feb '16

06 Feb '16

Maybe I'm totally overlooking something or misreading the docs, but I can't find a way to say in a requirements.txt file that a dependency is optional and its failure to install is okay. E.g., aiohttp supports using cchardet as an accelerator of chardet ( http://aiohttp.readthedocs.org/en/stable/#library-installation). I would like to be able to specify in my requirements.txt that I would like cchardet to be installed if possible, but it not being available -- which it might be as it doesn't have any Python 3.5 wheels ATM -- is fine and not the end of the world. I'm also happy to push a patch upstream if there is something aiohttp should be setting in their setup.py instead (I thought maybe extras, but those seem to be hard requirements as well).

3 2

Re: [Distutils] Fwd: setup('postinstall'='my.py')
by AltSheets Dev 06 Feb '16

06 Feb '16

Thank you Robert, Leonardo, Jeremy, Paul, Greg! > One very simple technique used by some projects like > numpy is just to have ``setup.py`` write a file into the > source tree before calling setup(). example: > https://github.com/numpy/numpy/blob/master/setup.py#L338-L339 Done that now. But: > The thing about setup.py is that it only runs at "build time". Yes, ouch. So ... not the way to go, in this case. Still good to know, useful for static but often updated version info, etc. > However, it looks like you want to generate a unique id at every install. Correct. The idea is to let me discover a possible abuse of serial keys - if a serial key appears together with many UUIDs, I can revoke it. (Not a protection against experts of course, but might still help.). My older frontend, a GoogleSheet (see altfolio.ddns.net) always automatically generates a UUID, in the moment when the user makes a copy of my mastersheet. That had inspired me now. Another goal of my questions on this list ... is learning about the whole process. I have been coding in Python for over a decade, but scientifically, sharing source code with other coders; or on a webserver - but I had never needed to make my code directly available to end users. I had always been wondering about packages & installers - now I am learning it. Your answers are great for me; *now* I understand all this much better. Thanks! > consider adding some code to the installer itself Wouldn't it be great if setuptools.setup provided that option, and OS-independent? > It appears that you are using bdist_wininst to create the installer. > So you are in luck! The bdist_wininst command supports running > simple scripts at a few points in the installation process. [...] > To use an installer script, you need to pass options to the > bdist_wininst command. '--pre-install-script=<pathname>' for the > pre-install script, where 'pathname' is the path to the script to > include (relative to setup.py or fully-qualified). > '--install-script=<scriptname>' for the post-install/pre-uninstall script Great. That sounds as if it is perhaps the way to go then, thanks a lot. Sounds as if it can do exactly what I need. However then ... aren't I stuck with a windows-only solution? Would it be a valid feature request to make those two useful options (pre-install-script/install-script) available platform-independent? > Here is (an untested) set of additions that should achieve > what you are asking for. [...] Hope this helps! Yes, fantastic. Very nice. Thanks a lot. Makes it 100% clear now. > > Does that mean that setup.py is then also not run during deployment > No. By the time a wheel is generated (or an egg, or any other bdist_* > command is run) the job of the setup.py for your package is done. That was the essential sentence that made "click" in my understanding - thanks. Might be a good sentence to be placed in a box in a central place in the manual. > I would consider the following strategies: > Create the UUID on first run (e.g. at import time) > Add a script (or function) for creating the UUID and document for your > users that it needs to be called before using your library > Have you thought of maybe not relying on the UID being generated at > "install" time, but rather having a "coinq init" command that the user > runs *once* to set up his/her UID? The other commands could then > check for that UID wherever you choose to store it, and if they don't > find it [...] automatically run init if they find no UID. Yes. I know, that is the other way to go. But then I need to understand where I am allowed to write persistent files - outside the place where the other .py files are installed anyways. And as that UUID.py has to be created only once in the lifetime of my tool on their computers, I had thought the /site-packages/... folder is the best place for it. But perhaps it is really not. Do you know a Python package that abstracts from the platform ... with which I can write to (would those be the right places?) (username/AppData/Roaming/myfolder) on Windows, and (~/.myfolder) on Linux - and is it (~/Library/myfolder) on Mac? Thanks a lot! :-)

3 2

version conflict
by Robin Becker 06 Feb '16

06 Feb '16

Hi, a user is trying to install reportlab and is getting an error I have not seen ie pkg_resources.VersionConflict: (certifi 2015.11.20.1 (/Library/Python/2.7/site-packages), Requirement.parse('certifi==2015.11.20')) the full traceback is visible here https://bitbucket.org/rptlab/reportlab/issues/73/versionconflict anyone here that can explain what's wrong? My pip installs of reportlab don't use any certifi packages etc etc. -- Robin Becker

2 1

wheel 0.27.0 released
by Nate Coraor 05 Feb '16

05 Feb '16

Hi all, I have just released wheel 0.27.0. This version includes a few new features and fixes for long standing minor issues, many from outside contributions. >From the changelog: 0.27.0 ====== - Support forcing a platform tag using `--plat-name` on pure-Python wheels, as well as nonstandard platform tags on non-pure wheels (Pull Request #60, Issue #144, thanks Andrés Díaz) - Add SOABI tags to platform-specific wheels built for Python 2.X (Pull Request #55, Issue #63, Issue #101) - Support reproducible wheel files, wheels that can be rebuilt and will hash to the same values as previous builds (Pull Request #52, Issue #143, thanks Barry Warsaw) - Support for changes in keyring >= 8.0 (Pull Request #61, thanks Jason R. Coombs) - Use the file context manager when checking if dependency_links.txt is empty, fixes problems building wheels under PyPy on Windows (Issue #150, thanks Cosimo Lupo) - Don't attempt to (recursively) create a build directory ending with `..` (invalid on all platforms, but code was only executed on Windows) (Issue #91) - Added the PyPA Code of Conduct (Pull Request #56) --nate

1 0