Mailman 3 July 2013 - Distutils-SIG

wheel without console_scripts in 0.20.0
by Daniel Holth 20 Jul '13

20 Jul '13

Hi Paul. https://bitbucket.org/dholth/wheel/commits/db913ecc2c0bb7fec14ba9eae68f99d9… Wheel 0.20.0 won't put entry_points console_scripts and gui_scripts in the wheel itself. The new "python -m wheel install-scripts wheel pip" command for example would install those scripts for wheel and pip into the default scripts directory. It requires setuptools, not just pkg_resources. Here's what I did to accomplish that using easy_install: dist = 'pip' pkg_resources_dist = pkg_resources.get_distribution(dist) install = wheel.paths.get_install_command(dist) command = easy_install.easy_install(install.distribution) command.args = ['wheel'] # dummy argument command.finalize_options() command.install_egg_scripts(pkg_resources_dist) It would be fun to experiment with alternative implementations like the "just import and run it" wrapper, and it might be nice to update the RECORD of installed files as part of the operation. It should also be possible to use less easy_install.

4 4

Specific packaging goals and a tentative timeline
by Nick Coghlan 20 Jul '13

20 Jul '13

We have a lot of initiatives going every which way at the moment, so I figured it would be a good idea to get a common perception of what we consider to be the important near term goals and a realistic timeline for improving the packaging ecosystem (in particular, the timing relative to the CPython 3.4 release cycle). One of the big things I'd like us to do is ensure we separate out "urgent" things that are coupled to the 3.4 release cycle (like ensuring in-place upgrades for pip work on Windows) from the "important but not urgent" things where we can take more time (like metadata 2.0) This is kinda long, but if people aren't used to long emails from me by now, they haven't been paying attention ;) (PyPA devs - please forward this to pypa-dev for discussion with the folks that don't frequent distutils-sig) My current impression is that the goals below should be fairly realistic. Already done or very close to done (Yay!): * improved PyPI SSL support * setuptools/distribute merger * easy_install SSL verification * setuptools support for additional hashes beyond md5 * pip 1.4 release with SSL verification and initial wheel support (soon!) Before Python 3.4 feature freeze (currently November 23, 2013) * decide on a bundling or explicit bootstrapping scheme for pip (this still needs a PEP to help clarify the pros and cons of the various alternatives) * get RM & installer builder consensus on that scheme * make any necessary updates to CPython (e.g. possibly adding Lib/getpip.py) * (hopefully) add support for indirect imports (see http://mail.python.org/pipermail/import-sig/2013-July/000645.html for the draft PEP - thanks Eric for taking this from a rough idea in email to a concrete proposal!) Before Python 3.4 first release candidate (currently Jan 18, 2014) * pip 1.5 available (or at least release candidates) * setuptools releases as needed * improved handling of in-place pip upgrades on Windows * improved handling of pip/setuptools/pkg_resources division of responsibility * both pip and setuptools available as cross platform wheel files on PyPI * Key requirement: "pip uninstall setuptools" must be supported & must not fundamentally break pip (but may disable installation from anything other than wheel files) * Highly desirable: possible to install pkg_resources without installing setuptools * Highly desirable: possible to install setuptools without the easy_install script (just the script, having the implementation in the setuptools.commands subpackage is fine) Following Python 3.4 final release (currently Feb 22, 2014) * further proposals target pip 1.6 - decoupled from CPython release cycle * metadata 2.0 (PEP 426/440) * sdist 2.0 and wheel 1.1 * installation database format update * revisit distlib-based pip (assuming 1.5 isn't based on a vendored distlib) * revisit TUF-for-PyPI (that's more likely to be pip 1.7 timeframe, though...) Independent activities & miscellaneous suggestions * maybe suggest "pip install distlib" over pip gaining its own programmatic API? * PEP 8 cleanup (including clarification of what constitutes an internal API) * improved PyPI upload API (Donald's working on this) * getting Warehouse to a point where it can be brought online as "pypi-next.python.org" * TUF-for-PyPI exploration (the TUF folks seems to have this well in hand) * improved local PyPI hosting (especially devpi) Specifically on the "bundle or bootstrap pip" front, I'll note that due to the concerns regarding how bundling pip with the CPython MSI installer may interact with in-place upgrades, I'm leaning back towards explicit bootstrapping, with an option to run the bootstrap as part of the installation process for both CPython 3.4+ and the Python Launcher for Windows. Doing that also gives Linux distros something they can patch in the system Python to direct users towards the appropriate system package manager command. Regardless, we still need the various bundling-or-bootstrap alternatives that aren't covered in PEP 439 extracted from the list archives and turned into a PEP that compares them and suggests a preferred solution. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

5 15

Upcoming changes to PEP 426/440
by Nick Coghlan 20 Jul '13

20 Jul '13

Donald has been continuing his data modelling work for Warehouse (aka PyPI 2.0: https://github.com/dstufft/warehouse) and found that the *_requires/*_may_require split for dependencies was significantly more painful to work with than I had expected. Accordingly, I'm making some adjustments to the way dependencies are defined to bring them more into line with the way the contributors field and contact metadata works: 1. The "*_may_require" fields are all going away (leaving only the "*_requires" fields) 2. The "*_requires" fields are becoming lists of "dependency specifier" mappings rather than strings 3. A dependency specifier is now a mapping with the following fields: * "install": the installation specifier for the dependency * "extra": as per the current PEP (for conditional dependencies) * "environment": as per the current PEP (for conditional dependencies) 4. The "install" subfield is compulsory, the other two are optional (as now, using either of the latter creates a "conditional dependency", while dependency declarations with only the "install" subfield are unconditional) 5. An installation specifier is what PEP 426 currently calls a dependency specifier: the "name [extras] (constraints)" format. They will get their own top level section (similar to the existing Extras and Environment markers sections) In addition to those changes, I'll also be making a change to the recommended handling of virtual distributions (again based on Donald's feedback): integration tools should NEVER look at the "Provides" listings on a public index server to satisfy unmet dependencies. Instead, virtual distributions should be defined in such a way that whoever first invents the specific virtual distribution name registers it on PyPI, using the dependency metadata to pull in a default implementation. That's the only way to manage virtual distributions on a public index server that isn't vulnerable to later hijacking simply by registering a distribution with that name. As part of documenting that, I'll probably give the notion of "Virtual distributions" their own top level section (these are distributions that don't have any code of their own - they just declare dependencies on other projects to make them easy to install as a group, or to define the default provider for a dependency that may be satisfied by any one of multiple distributions). Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

5 40

distlib Wheel.install API - proposed changes - distlib users, please comment
by Vinay Sajip 19 Jul '13

19 Jul '13

The API for installing wheels in distlib [1] will need changes as a result of recent discussions around the need to have better control over how scripts are generated at installation time. Currently, the API looks like this: def install(self, paths, dry_run=False, executable=None, warner=None, lib_only=False): """ Install a wheel to the specified paths. If ``executable`` is specified, it should be the Unicode absolute path the to the executable written into the shebang lines of any scripts installed. If ``warner`` is specified, it should be a callable, which will be called with two tuples indicating the wheel version of this software and the wheel version in the file, if there is a discrepancy in the versions. This can be used to issue any warnings to raise any exceptions. If ``lib_only`` is True, only the purelib/platlib files are installed, and the headers, scripts, data and dist-info metadata are not written. The return value is a :class:`InstalledDistribution` instance unless ``lib_only`` is True, in which case the return value is ``None``. """ Internally, this method constructs a ScriptMaker [2] instance and uses it to write the scripts, any variants of the type fooX and foo-X.Y, and any native executable wrappers, to the installation target directory. Currently, scripts defined in exports (the equivalent of setuptools' "entry points") are written into the wheel at build time, and just copied to the target installation directory in this method. This is likely to change to require script generation as well as copying to happen at installation time. This means that we need to pass additional information to the install method, but the API already has quite a few keyword arguments, and it seems unwise to add any more. The "executable" argument is just passed to the ScriptMaker, and not otherwise used. So I propose to stream the API to be as follows: def install(self, paths, maker, options): The dry_run will be taken from the maker instance. The caller will be responsible for instantiating the maker and configuring it (e.g. setting a custom executable for shebangs, enabling fooX/foo-X.Y variants, etc). The warner and lib_only will be obtained from the options argument, which is just a holder for values in attributes, using warner = getattr(options, 'warner', None) lib_only = getattr(options, 'lib_only', False) Although this approach hides the individual options from the signature, which I don't like, it will avoid adding more arguments to the signature, which I do like. With PEP 426 still somewhat fluid, better controllability might require more options to be added. Does anyone have any views about this, or any suggestions for improvement? In the absence of any feedback, I'll make these changes soon. They will of course break any code you have which uses this API, hence the call for comments. Regards, Vinay Sajip [1] https://bitbucket.org/pypa/distlib/src/tip/distlib/wheel.py?at=default#cl-3… [2] https://bitbucket.org/pypa/distlib/src/tip/distlib/scripts.py?at=default#cl…

1 0

API for registering/managing URLs for a package
by M.-A. Lemburg 19 Jul '13

19 Jul '13

I would like to write a script to automatically register release URLs for PyPI packages. Is the REST API documented somewhere, or is the implementation the spec ? ;-) And related to this: Will there be an option to tell PyPI's CDN to cache the release URL's contents ? Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

3 4

"ImportError: No module named setuptools" (when using "pip install --upgrade")
by Marcus Smith 18 Jul '13

18 Jul '13

If you're getting "ImportError: No module named setuptools" when using "pip install --upgrade", see here for an explanation and solution: https://github.com/pypa/pip/issues/1064

1 0

Worry about lack of focus
by Noah Kantrowitz 18 Jul '13

18 Jul '13

So we've recently seen a big resurgence in activity on improving Python packaging. First off, thats good, hopefully thats why we are all here. That said, I'm becoming worried about a possible lack of focus, and I know I'm not the only one. There have been many ideas floated, and many PEPs either sketched out, reworked, or are stated to be in planning. I think perhaps we should work out some kind of shortlist of what we think can and should be accomplished in the short term and just keep a running list of topics that need energy but are lower priority. This would reduce the chances of hitting the "fix the whole world at once" situation that we have run in to before in this attempt, which often results in burnout and frustration all around. Just to kick things off here are the rough topics I can think of that I've seen discussed recently (ignoring that many of these are dependent on each other): * Including pip with Python 3.4 * Bundling setuptools with pip * Splitting setuptools and pkg_resources * Replacing the executable generation in pip with something new * Working out how to let pip upgrade itself on Windows * Entrypoints in distutils/the stdlib * Executable generation in distlib * Signing/vetting of releases * General improvements to the wheel format * General improvements to package metadata Apologies for anything I have mis-paraphrased or missed, but that is definitely a lot of things to have up in the air. Just want to make sure we can get everything done without anyone going crazy(er) and that we keep sight of whats going on. --Noah

4 4

Replacing pip.exe with a Python script
by Paul Moore 18 Jul '13

18 Jul '13

I'm looking at the possibility of replacing the current setuptools entry point based pip executables with Python scripts. The biggest problem is that a script "pip.py" shadows the pip package, making "import pip" fail. I can get round this by deleting sys.path[0] (the location of the currently running script) but how robust is that? Are there any corner cases where it would break? Or alternatively, is there a better way to do this rather than messing directly with sys.path? I suspect this is a fairly common question, but my Google-fu is failing me :-( Sorry, I know this is a basic Python coding question - in my defence, it's for something related to the current pip discussions :-) Paul

11 27

distribute 0.7.3 causing installation error?
by Liam Kirsher 18 Jul '13

18 Jul '13

2 3

vetting, signing, verification of release files
by holger krekel 18 Jul '13

18 Jul '13

I am considering implementing gpg-signing and verification of release files for devpi. Rather than requiring package authors to sign their release files, i am pondering a scheme where anyone can vet for a particular published release file by publishing a signature about it. This aims to help responsible companies to work together. I've heart from devops/admins that they manually download and check release files and then install it offline after some vetting. Wouldn't it be useful to turn this into a more collaborative effort? Any thoughts or pointers to existing efforts within the (Python) packaging ecologies? best, holger

11 24