Mailman 3 March 2016 - Distutils-SIG

abstract build system approaches redux
by Robert Collins 04 Apr '16

04 Apr '16

This is my attempt to consolidate the three different proposals we have at the moment into a single set of design choices. We have three draft designs - from Nathaniel (517), Donald (unnumberd) and mine (516). For purposes of comparison, I've skipped all rationale here, and only listed the highlights that differ across the proposals. If you haven't read all three, you should before continuing. My goal is to let us pick a final shape - and then we can shave the rough edges off of that shape to get a final PEP (or PEPs) 517: - Python interface to the system - build requires, optional wheel metadata (to directory on disk), build wheel. install editable commands 516: - CLI interface to the system - build requires, mandatory wheel metadata, build wheel, install editable commands Donald's: - Python interface to the system - defines new 'source wheel' concept - source_metadata, source_wheel, binary_metadata, binary_wheel commands - 'develop' // editable not addressed currently - outputs directories on disk, rather than zipped up things Ok, now here's my view on the differences... Python vs CLI - I don't care enough to argue. I think a CLI is better in this space, as running commands is the lingua franca of CLI's; previously Donald has advocated for that as well, but his proposal is now a Python API. *shrug*. I want the BDFL-Delegate to choose. 516 and 517 are otherwise identical in all meaningful ways - there's plenty of stuff to bikeshed on (e.g. should wheel metadata be optional or mandatory), but I don't think it is crucial: we can always change this in future by issuing a new version. Its not free, but if we can't agree, we can at least find out later. Donalds and 516 both have the ability to get metadata out explicitly as a mandatory thing, so I think we should make it mandatory and weaken it in future if e.g. flit eventually come back saying its a burden. [I don't believe it will be]. File formats - BDFL-Delegate can pick. I so don't care :). New source format. I'm quite strongly against this at the moment: Building from VCS is now a fairly major thing in the community, and we can't mandate new URLs for all sources. We're going to thus be forcing a change to all tooling (e.g. Debian and Fedora and pip and others) that know how to build things from an arbitrary VCS url. A new source extension in pypi doesn't reduce the impact of that at all, and the necessary code to handle that will also handle sdists's on pypi that have the new config file and may be missing a setup.py. If there is a new source extension in pypi, we should expect that folk using e.g. flit will *not* upload old style sdists to pypi - they don't at the moment, and the objection to doing so will *remain*. -> We don't save any grief or friction for the adoption of the new thing, nor enable new build systems any more easily. A shim to the new shiny would enable current sdists for anything adopting the new thing in exactly the same manner as it would if we reuse the extension. I can see an argument for a new iteration of the sdist format for better metadata, and to be able to tell what metadata is trustable etc, but I don't see that as related to the problem of enabling third party build systems - and since all the easy cases will be solved by binary wheels I think we're in rapidly diminishing territory here - if something can't upload a binary wheel, its quite likely to have complex dependencies - and we haven't solved the numpy ABI problem yet, so I'd like to actually focus on that well and solve it - and once solved see if we can retrofit it into the existing sdist format, or if a new one is actually needed. Develop is essential to tackle IMO. We can either fully PEP define it now - and if so we should stop talking about this and start talking about that. If we don't have bandwidth to do that yet (and AFAICT we don't, based on IRC discussions and the headaches of interactions with the three different namespace package possibilities, virtualenvs and user installs....). Outputting the wheel as an unzipped directly on disk would be a nice optimisation for big trees, I think, so perhaps we should apply that to 516/517 to reduce the set of things we're discussing. -Rob 516: https://github.com/pypa/interoperability-peps/blob/master/pep-0516-build-sy… 517: https://github.com/pypa/interoperability-peps/blob/master/pep-0517-build-sy… Donalds: https://github.com/pypa/interoperability-peps/compare/master...dstufft:buil… -- Robert Collins <rbtcollins(a)hpe.com> Distinguished Technologist HP Converged Cloud

5 8

Thank you for the ability to do `pip install git+https://...`
by Nicholas Chammas 01 Apr '16

01 Apr '16

Dunno how old/new this feature is, or what people did before it existed, but I just wanted to thank the people who thought of and built the ability to do installs from git+https. It lets me offer the following to my users when they want the “bleeding edge” <https://github.com/nchammas/flintrock#development-version> version of my project: pip install git+https://github.com/nchammas/flintrock I also use this capability to install and test contributors’ branches when they open PRs against my project. For example: pip install git+https://github.com/contributor/flintrock@branch It’s a great feature and makes my work a bit easier. Thank you for building it. I’m still waiting for when I can give the PyPA some money <https://github.com/pypa/warehouse/issues/856> for all the good and sorely needed work that y’all do… Anyway, keep it up. Nick

4 4

question re: local development workflow / egg-info / synced directories
by Chris Jerdonek 31 Mar '16

31 Mar '16

Hi, I have a local development workflow question I'm looking for feedback or suggestions on. It relates to installing things in editable mode in conjunction with syncing source files to a VM. I'm developing a Python project that is deployed / runs inside a Docker container. I do my local development on a Mac, and I use the following script to "sync" my local source files to the Docker container: https://github.com/brikis98/docker-osx-dev At Docker build time, the project source files are installed in "editable" mode, which creates egg-info directories inside the source directories in the Docker image. When developing locally, however, the sync process mounts the synced directory "over" the directory that was initialized at Docker-build time. Thus the egg-info directories are missing from the synced version of the directory. Does anyone have any suggestions for handling this scenario? The three possibilities I can think of are-- 1. re-install the projects in the Docker container in editable mode after mounting the synced directory (which is a bit hacky because there is no obvious "hook" with the tools I'm using), 2. install the projects in editable mode locally on my Mac so the egg-info directories will also sync over (but I'm not sure if egg-info directories are sufficiently cross-platform), or 3. maybe there is a way to install things in "editable" mode that doesn't require egg-info directories to be written alongside the source files? Does anyone have any suggestions? And does anyone know if (3) is possible? Thanks a lot for any help, --Chris

3 7

Processing dependencies on Gihtub
by Demian Brecht 26 Mar '16

26 Mar '16

Hi all, I'm investigating the ability to replace PyPI with Github-based mirrors. The use case is that there is no outgoing internet access from a set of hosts responsible for building Python packages. However, there /is/ access to a Github enterprise instance. Installing top level packages through git+https works great, but (unless I'm wrong) falls down when handling dependencies. I've looked at --find-links, which doesn't seem to do the trick (I'm assuming it's expecting a PyPI-compatible index). I've also read the --process-dependency-links is deprecated and will eventually be removed. Is that correct? If that's all accurate, are there any suggested paths to install top level packages and their respective dependencies solely using Github? Are there any API or HTML pages that I'm unaware of that would conform to pip requirements? The only avenue that I can think of is to write a small proxy service that simply presents a format understood by pip. One of the things that I'm trying to avoid is touching mirrored projects. Not all projects use requirements files and obviously none of them will be using the internal GHE instance that I'm looking to use as a home for mirrored projects. Thanks, ------- GPG Fingerprint: 9530 B4AF 551B F3CD A45C 476C D4E5 662D DB97 69E3

2 1

PEP 425 vs pip
by David Cournapeau 23 Mar '16

23 Mar '16

Hi, At Enthought we have been trying to use PEP 425 tags to include its metadata in our packages. It has been harder than expected because of missing information in PEP 425, and maybe inconsistencies with what pip actually does. I think PEP 425 should be updated to take into account pip actual implementation: 1. python tag and pypy: the PEP says "the version is py_version_nodot . CPython gets away with no dot, but if one is needed the underscore _ is used instead. PyPy should probably use its own versions here pp18 , pp19 ". This is not what pip does as of version 8.1.0 for pypy. Instead, the version is python major + sys.pypy_version_info.major + sys.pypy_version_info.minor. 2. It would be nice to know how to deal with other alternative implementations as well. Would the scheme in 1. work for jython, etc... ? 3. The abi tag should at least be specified for cpython IMO. E.g. the meaning of `m`, `u` and `d` should be clearly specified. While the PEP mentions each implementation community is welcome to use the abi tag as they wish, given that in practice it is defined in pip, I think that should be specified as well. Do the above make sense ? If so, I could spend some time to update it (or create a new amending PEP ?) thanks, David

2 3

Python on AIX 7.1
by Gene Fontanilla 21 Mar '16

21 Mar '16

Can anyone guide me in installing Python AIX? i have no experience in AIX. Thanks and Regards,

2 1

python_version>="2.7" environment marker difficulties
by Michael R. Crusoe 21 Mar '16

21 Mar '16

Hello, To support a dependency[0] which has a different name for its Py2 & Py3 versions, our program schema_salad uses the "python_version" environment marker. This seems to not be supported when users have older setuptools installed[1] With setuptools before version 17.1 one gets error in schema-salad setup command: Invalid environment marker: python_version>="2.7" I tried to bundle a newer version with ez_setup.py but that isn't working either: https://github.com/common-workflow-language/schema_salad/commit/04a956dbef4… "ez_setup() bootstraps transparently if you're using python setup.py install but not for pip install of the same package (where it complains that you need to upgrade setuptools first)" Code is at https://github.com/common-workflow-language/schema_salad/blob/envmarker-tes… Is there a better way to deal with the divergent names for our dependency? Thanks in advance, [0] https://pypi.python.org/pypi/avro vs https://pypi.python.org/pypi/avro-python3 [1] https://github.com/common-workflow-language/schema_salad/issues/17#issue-14… -- Michael R. Crusoe CWL Community Engineer crusoe(a)ucdavis.edu Common Workflow Language project University of California, Davis https://impactstory.org/MichaelRCrusoe http://twitter.com/biocrusoe

2 1

What's up with PyPi maintenance?
by Chris Barker 18 Mar '16

18 Mar '16

Probably not the right list but: There are a number of folks having issue siwth new PyPi pacakges not being found by search: https://bitbucket.org/pypa/pypi/issues/412/my-package-doesnt-show-up-in-sea… clearly issues have been posted to the project, but has been going on a for a while, with seemingly no action, or response from PyPi developers/maintainers. a) Is there somewhere else we can ping the right fols? b) Is there a serious lack of folks available to address such issues? if (b), should we make a concerted effort to recruit new folks to assist PyPi maintenance / development??? Thanks, -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

7 9

Wheels and Python implementation markers
by Alex Grönholm 18 Mar '16

18 Mar '16

What is the "correct" way to specify (for building wheels) in setup.py that a dependency should only be installed on CPython? If so, how should I go about it? Nothing I've tried seems to work right. The python_implementation marker worked up until packaging 16.5 and then platform_python_implementation does not work with older versions of pip.

1 0

Build systems & working directories
by Nick Coghlan 14 Mar '16

14 Mar '16

I'm currently trying out flit for a new project, and find it nicely simple to use. However, I've also found a current constraint in terms of the source tree layout: if you use a repo structure with a top level "src" directory, then the flit.ini file needs to be in that directory. This means that in PEP 516 terms, the build command will need to be "flit -f src/flit.init" rather than just a plain "flit". I think the PEP could stand to be a bit more explicit that when the specified build_command is invoked, the current working directory will be the root of the source tree. (The PEP sort of implies this already, but I didn't see it explicitly stated anywhere) Regards, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0