Mailman 3 August 2013 - Distutils-SIG

State of the wheel spec (Was: How to handle launcher script importability?)
by Paul Moore 22 Aug '13

22 Aug '13

On 21 August 2013 13:01, Paul Moore <p.f.moore(a)gmail.com> wrote: > If you want to say distlib won't support pre-Metadata 2.0 specifications > of script metadata, then that's your choice - it's not contrary to the > standards but I'd view it as a quality of implementation choice. I view the > underspecification in the Wheel 1.0 spec as similarly a quality of detail > issue, and I'd expect to fix it in either an update to Wheel 1.0, or a > Wheel 1.1 which does not make the jump to pure Metadata 2.0. Ignore this (to an extent). I've just seen your comments on the distlib issue. For those not following along, both wheel and distlib currently process a pydist.json file in the wheel dist-info directory. The wheel project doesn't put entry-points.txt data in there yet, but distlib expects to find script metadata there. So the incompatibility here is merely one of two implementations being out of sync - no big deal. BUT, this means that there is no spec of the current behaviour, and no implementation of the Wheel 1.0 spec anywhere. Either there is movement behind the scenes that I'm not aware of, and I'm jumping the gun in trying to make sense of the current state of play, or the wheel spec needs a review reasonably soon. Whichever is the case, I don't think that my current investigations are adding anything productive here. So I'm going to take a step back from all of this for a week or two and see what develops. Paul

4 8

distutils.util.get_platform() - Linux vs Windows
by samuel.ferencik＠barclays.com 22 Aug '13

22 Aug '13

Hi, It seems distutils.util.get_platform() semantically differs on Windows and Linux. Windows: the return value is derived from the architecture of the *interpreter*, hence for 32-bit Python running on 64-bit Windows get_platform() = 'win32' (32-bit). Linux: the return value is derived from the architecture of the *OS*, hence for 32-bit Python running on 64-bit Linux get_platform() = 'linux-x86_64' (64-bit). Is this intentional? My context (where this hit me): I was trying to install the 32-bit version of the Perforce API (compiled module) on 64-bit Windows and on 64-bit Linux. My command-line was python3.3-32 setup.py install --root FOO --install-platlib=lib.$PLAT (note the '-32' and the '$PLAT') On Windows, this installed the 32-bit version of the API into "FOO\lib.win32". On Linux, this installed the 64-bit version of the API into "FOO/lib.linux-x86_64". I'm not sure which is more correct behavior but the asymmetry seems suspicious. Sam _______________________________________________ This message is for information purposes only, it is not a recommendation, advice, offer or solicitation to buy or sell a product or service nor an official confirmation of any transaction. It is directed at persons who are professionals and is not intended for retail customer use. Intended for recipient only. This message is subject to the terms at: www.barclays.com/emaildisclaimer. For important disclosures, please see: www.barclays.com/salesandtradingdisclaimer regarding market commentary from Barclays Sales and/or Trading, who are active market participants; and in respect of Barclays Research, including disclosures relating to specific issuers, please see http://publicresearch.barclays.com. _______________________________________________

4 9

Installing from a wheel
by Oscar Benjamin 21 Aug '13

21 Aug '13

This is the first time that I've tested using wheels and I have a couple of questions. Here's what I did (is this right?): $ cat spam.py # spam.py print('running spam from:', __file__) $ cat setup.py from setuptools import setup setup(name='spam', version='1.0', py_modules=['spam']) $ python setup.py bdist_wheel running bdist_wheel ... creating build\bdist.win32\wheel\spam-1.0.dist-info\WHEEL $ ls build dist setup.py spam.egg-info spam.py $ ls dist/ spam-1.0-py27-none-any.whl Okay, so far so good. I have the wheel and everything makes sense. Now I want to test installing it: $ wheel install --wheel-dir=./dist/ spam The line above gives no output. I expect something like 'installing spam... installed.'. It also ran so quickly that I thought that nothing had happened. A quick check reveals that the module was installed: $ cd ~ $ python -m spam ('running spam from:', 'q:\\tools\\Python27\\lib\\site-packages\\spam.py') $ pip list | grep spam spam (1.0) So now how do I uninstall it? $ pip uninstall spam Can't uninstall 'spam'. No files were found to uninstall. The wheel command doesn't seem to have an uninstall option either. Oscar

3 10

What does it mean for Python to "bundle pip"?
by Paul Moore 19 Aug '13

19 Aug '13

While debate on PEP 439 has died down, I've been thinking about what it actually means to end users for Python to bundle pip. If we have an explicit bootstrapping command like get_pip, how much practical benefit is there for that to be bundled vs having a well-known location for such a script (which is more or less what we have now)? Also, there is the whole question of how much we want to "lock down" what pip currently provides, as official functionality. (Particularly given the renewed energy around projects like distlib/distil, etc). I'd like to propose that as a first step, we agree and document a *user interface* that will be officially supported going forward. It would be based on (a subset of) the pip interface, so that a "bundle pip" strategy is still the immediate way forward, but by standardising the interface rather than the implementation, we retain the option to change things behind the scenes if, for example, distil suddenly takes over the world. Users get the benefit of a properly sanctioned "one true way" to manage packages. And we reserve enough flexibility that we don't accidentally accept constraints that are stronger than we'd like and stifle innovation. To be specific, what I propose is that we agree on the following: 1. Python installations provide a bootstrap command (tentatively "python -m get_pip", I'm not sure this needs to be directly executable, the -m interface is probably fine). This command may be a no-op if for example we opt for full bundling (and it will always be a no-op after the first run). 2. Once bootstrapped, Python will provide a "pip" command (or pip3?) accessible whenever the "python" command is available (worded this way because the Windows installer doesn't put "python" on PATH by default). 3. A specific subset of the current pip interface is provided (to be documented). I'd suggest we need a minimum of pip install, pip list, pip uninstall, pip install -U (for upgrades). I don't know what proportion of the rest of the pip API (various options, requirement files, environment variables and ini files) we want to lock down - I suggest we start minimal and add items as needed. 4. We may want to add a separate note that "python -m pip" will do the same as the "pip" command, and may be needed to self-upgrade the pip command ("python -m pip -U pip"). We can note for 3.4 that the pip command will be implemented using the current pip project, and so all of the other features of pip will be available, but with the proviso that any interfaces not explicitly mentioned in the standard documentation may be changed or removed as pip changes, and are not protected by Python's compatibility rules. That gives users a reasonable level of stability, while still allowing us the room to innovate. Does this sound reasonable? It may be obvious to everyone but me that this is what we're expecting - or I may even be proposing *more* than people expect. But I think something along the lines of an "interface over implementation" definition like this would be reasonable. If this is a useful proposal and no-one else is already working on something similar, I'm willing to write this up as a PEP. Paul.

4 12

python version problem during install distutils
by NAITIK PATEL 19 Aug '13

19 Aug '13

Hi I wan to install distutils modul on python2.5 for that I downloaded Distutils-1.0.2. But during the installation it make some problem. give some error. File "setup.py", line 30, in <module> packages = ['distutils', 'distutils.command'], File "/tmp/Distutils-1.0.2/distutils/core.py", line 101, in setup _setup_distribution = dist = klass(attrs) File "/tmp/Distutils-1.0.2/distutils/dist.py", line 130, in __init__ setattr(self, method_name, getattr(self.metadata, method_name)) AttributeError: DistributionMetadata instance has no attribute 'get___doc__' please help me for that Regards, Naitik Patel

2 1

Looks like pypi is down, I'm getting 500 errors...
by Paul Moore 19 Aug '13

19 Aug '13

The CDN seems OK (pip install works fine) but the backend (search, etc) looks like it has problems... Paul

1 0

Changing the "install hooks" mechanism for PEP 426
by Nick Coghlan 16 Aug '13

16 Aug '13

I spent last weekend at "Flock to Fedora", mostly due to my day job working on the Beaker integration testing system (http://beaker-project.org) for Red Hat, but also to talk to folks about Fedora and Python interactions. A completely unexpected discovery over the weekend, was that some of the RPM folks are exploring the idea of switching the *user* facing format for the packaging system away from spec files and towards directly executable Python code. Thus, you'd get away from the painful mess that is RPM conditionals and macros and have a real programming language to define what your built packages *should* look like, while still *producing* static metadata for consumption by installers and other software distribution tools. Hmm, does that approach sound familiar to anyone? :) Anyway, we were talking about how they're considering approaching the install hook problem, and their approach gave me an idea for a better solution in PEP 426. Currently, PEP 426 allows a distribution to define "install hooks": hooks that will execute after the distribution is installed and before it is uninstalled. I'm now planning to change that to allowing distributions to define "export hooks", based on the cleaned up notion of "export groups" in the latest version of PEP 426. An export hook definition consists of the following fields: * group - name of the export group to hook * preupdate - export to call prior to installing/updating/removing a distribution that exports this export group * postupdate - export to call after installing/updating/removing a distribution that exports this export group * refresh - export to call to resynchronise any caches with the system state. This will be invoked for every distribution on the system that exports this export group any time the distribution defining the export hook is itself installed or upgraded If a distribution exports groups that it also defines hooks for, it will exhibit the following behaviours: Fresh install: * preupdate NOT called (hook not yet registered) * postupdate called * refresh called Upgrade: * preupdate called (old version) * postupdate called (new version) * refresh called (new version) Complete removal: * preupdate called * postupdate NOT called (hook no longer registered) * refresh NOT called (hook no longer registered) This behaviour follows naturally from *not* special casing self-exports: prior to installation, the export hooks won't be registered, so they won't be called, and the same applies following complete removal. The hooks would have the following signatures: def preupdate(current_meta, next_meta): # current_meta==None indicates fresh install # next_meta==None indicates complete removal def postupdate(previous_meta, current_meta): # previous_meta==None indicates fresh install # current_meta==None indicates complete removal def refresh(current_meta): # Used to ensure any caches are consistent with system state # Allows handling of previously installed distributions Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

4 18

tox-1.6: install_command, develop, py25 support, json-reporting ...
by holger krekel 15 Aug '13

15 Aug '13

tox-1.6: support for install_command, develop, json-reporting ============================================================= Welcome to a new release of tox, the virtualenv-based test automation manager. This release brings some new major features: - installer_command: you can customize the command user for installing packages and dependencies. Thanks Carl Meyer. - usedevelop: you can use "develop" mode ("pip install -e") either by configuring it in your tox.ini or through the new "--develop" option. Thank Monty Tailor. - python2.5: tox ships internally virtualenv-1.9.1 and can thus run tests create virtualenvs and run tests against python2.5 even if you have a newer virtualenv version installed. While tox-1.6 should otherwise be compatible to tox-1.5, the new $HOME-isolation ($HOME is set to a temporary directory when installing packages) might trigger problems if your tests relied on $HOME configuration files -- which they shouldn't if you want to repeatability. If that causes problems, please file an issue. Docs and more information at: http://tox.testrun.org/tox/latest/ have fun, holger 1.6 Changelog -------------- - fix issue35: add new EXPERIMENTAL "install_command" testenv-option to configure the installation command with options for dep/pkg install. Thanks Carl Meyer for the PR and docs. - fix issue91: python2.5 support by vendoring the virtualenv-1.9.1 script and forcing pip<1.4. Also the default [py25] environment modifies the default installer_command (new config option) to use pip without the "--pre" option which was introduced with pip-1.4 and is now required if you want to install non-stable releases. (tox defaults to install with "--pre" everywhere). - during installation of dependencies HOME is now set to a pseudo location ({envtmpdir}/pseudo-home). If an index url was specified a .pydistutils.cfg file will be written with an index_url setting so that packages defining ``setup_requires`` dependencies will not silently use your HOME-directory settings or https://pypi.python.org. - fix issue1: empty setup files are properly detected, thanks Anthon van der Neuth - remove toxbootstrap.py for now because it is broken. - fix issue109 and fix issue111: multiple "-e" options are now combined (previously the last one would win). Thanks Anthon van der Neut. - add --result-json option to write out detailed per-venv information into a json report file to be used by upstream tools. - add new config options ``usedevelop`` and ``skipsdist`` as well as a command line option ``--develop`` to install the package-under-test in develop mode. thanks Monty Tailor for the PR. - always unset PYTHONDONTWRITEBYTE because newer setuptools doesn't like it - if a HOMEDIR cannot be determined, use the toxinidir. - refactor interpreter information detection to live in new tox/interpreters.py file, tests in tests/test_interpreters.py.

1 0

devpi-1.0: improved PyPI-server with upload/test/staging tool
by holger krekel 15 Aug '13

15 Aug '13

devpi-1.0: PyPI server and packaging/testing/release tool ========================================================= devpi-1.0 brings an improved PyPI caching and internal index server as well as a new abilities for tox-testing and staging your Python release packages. For a (long) list of changes, see the below CHANGELOG. Documentation got revamped and extended and now contains three quickstart scenarios. First the Quickstart tutorial for pypi-mirroring on your laptop:: http://doc.devpi.net/1.0/quickstart-pypimirror.html And if you want to manage your releases or implement staging as an individual or within an organisation:: http://doc.devpi.net/1.0/quickstart-releaseprocess.html If you want to permanently install devpi-server and potentially access it from many clients:: http://doc.devpi.net/1.0/quickstart-server.html More documentation and the beginning of an exhaustive user manual:: http://doc.devpi.net/latest/ Note that devpi-1.0 is not data-compatible to the previous 0.9.4 release: You need to start with a fresh devpi-1.0 installation and upload your packages again. Future releases of devpi should support data migration more directly. best and have fun, holger krekel Changelog 1.0 (-0.9.4) ---------------------------- devpi-server: - rename "--datadir" to "--serverdir" to better match the also picked up DEVPI_SERVERDIR environment variable. - fix a strange effect in that sometimes tools ask to receive a package url with a "#md5=..." arriving at the server side. We now strip that part out before trying to serve the file. - on startup don't create any initial indexes other than the "root/pypi" pypi caching mirror. - introduce ``--start``, ``--stop`` and ``--log`` commands for controling a background devpi-server run. (these commands previously were implemented with the devpi-client and the "server" sub command) - fix issue27: provide full list of pypi names in root/pypi's simple view (and simple pages from inheriting indices) - default to "eventlet" server when creating deployment with --gendeploy - fix issue25: return 403 Forbidden when trying to delete the root user. - fix name mangling issue for pypi-cache: "project_name*" is now matched correctly when a lookup for "project-name" happens. - fix issue22: don't bypass CDN by default, rather provide an "--bypass-cdn" option to do it (in case you have cache-invalidation troubles) - fix issue20 and fix issue23: normalize index specs internally ("/root/dev" -> "root/dev") and check if base indices exist. - add Jenkins build job triggering for running the tests for a package through tox. - inheritance cleanup: inherited versions for a project are now shadowed and not shown anymore with getreleaselinks() or in +simple pages if the "basename" is exactly shadowed. - fix issue16: enrich projectconfig json with a "+shadow" file which lists shadowed "versions" - initial wheel support: accept "whl" uploads and support caching of whl files from pypi.python.org - implemented internal push operation between devpi indexes - show "docs" link if documentation has been uploaded - pushing releases to pypi.python.org will now correctly report the filetype/pyversion in the metadata. - add setting of acl_upload for indexes. Only the owning user and acl_upload users may upload releases, files or documentation to an index. - add --passwd USER option for setting a user's password server-side - don't require email setting for creating users devpi-client: - removed ``server`` subcommand and options for controling background devpi-server processes to become options of ``devpi-server`` itself. - fix issue14: lookup "python" from PATH for upload/packaging activities instead of using "sys.executable" which comes from the interpreter executing the "devpi" script. This allows to alias "devpi" to come from a virtualenv which is separate from the one used to perform packaging. - fix issue35: "devpi index" cleanly errors out if no index is specified or in use. - remember authentication on a per-root basis and cleanup "devpi use" interactions. This makes switching between multiple devpi instances more seemless. - fix issue17: better reporting when "devpi use" does not operate on valid URL - test result upload and access: - "devpi test" invokes "tox --result-json ..." and uploads the test result log to devpi-server. - "devpi list [-f] PKG" shows test result information. - add "uploadtrigger_jenkins" configuration option through "devpi index". - fix issue19: devpi use now memorizes --venv setting properly. Thanks Laurent. - fix issue16: show files from shadowed versions - initial wheel support: "devpi upload --format=bdist_wheel" now uploads a wheel format file to the index. (XXX "devpi install" will trigger pip commands with option "--use-wheels".) - fix issue15: docs will now be built via "setup.py build_sphinx" using a internal build dir so that the upload succeeds if conf.py would otherwise specify a non-standard location. - implement and refine "devpi push" command. It now accepts two forms "user/name" for specifying an internal devpi index and "pypi:REPONAME" for specifying a repository which must be defined in a .pypirc file. - remove spurious pdb.set_trace() in devpi install command when no pip can be found. - show and allow to set "acl_upload" for uploading priviliges - add longer descriptions to each sub command, shown with "devpi COMMAND -h". - removed pytestplugin support for now (pytest reporting directly to devpi-server)

1 0

Realistic PyPI, pip and TUF demo
by Trishank Karthik Kuppusamy 15 Aug '13

15 Aug '13

Hello everyone, We now have a demonstration of pip that securely and efficiently downloads with TUF any package from a PyPI mirror: https://github.com/theupdateframework/pip/wiki/pip-over-TUF We hope that you will try our demonstration with your favourite packages and tell us about any issue that you find. TUF does not yet work on Microsoft Windows and Apple OS X. This is because it depends for cryptography on a custom Python library (evpy) which binds with OpenSSL. We are planning to fix this by moving to the cross-platform Mozilla Network Security Services (NSS) library. We also welcome your thoughts on features and enhancements that you would like to see. Our next demo will show security flaws in package managers such as pip that do not use TUF. We will then see how pip with TUF addresses those security attacks. -The TUF team

1 0