Mailman 3 June 2013 - Distutils-SIG

PEP 440 and direct references
by Vinay Sajip 27 Jun '13

27 Jun '13

I would find it helpful to understand the motivation for direct references as defined in PEP 440, and some clarifications of how they can be used. For example: * Can they be used together with other clauses? As PEP 440 appears now, it seems to indicate that they can, but what does it mean in practice to have a clause which is so specific with other clauses which are less specific? * Can we have information in the "Direct reference" section summarising where such references can and can't appear? * What direct references provide which cannot be provided by the other version clauses together with (separately) URIs or paths for where archives can be found? The fact that the format of this clause is so different from all the others suggests that it doesn't really belong with them. There are also questions on other areas: * It would appear that explicit prefix matching only makes sense for == and != clauses (as it is implicit for e.g.<, > clauses). If so, can this be stated explicitly? * Since pre-/post-/dev- suffixes are ignored for the purposes of prefix matching, and given that implicit prefix matching is used with ~= , < and >, does this mean that explicit prefix matching is only useful for >=, <=? (given that == with prefix matching appears equivalent to ~=) It would also be useful to document what kinds of matches would not be possible without ~= and explicit prefix matching (trailing .*), assuming we had implicit prefix matching for ==/!= clauses. Regards, Vinay Sajip

3 8

Important! buildout 2.1.1 released, prepare for distribute 0.7 and buildout 2.2
by Jim Fulton 26 Jun '13

26 Jun '13

Distribute/setuptools 0.7 bring some big changes. These changes can't be handled by buildout's normal automatic upgrade mechanism. For this reason, it's important to do one of the following if you're using buildout 2: - Pin your distribute version to <0.7dev or to some other 0.6 version. - Upgrade to buildout 2.1.1, which was just released. This will prevent automatic upgrade to distribute 0.7 or buildout 2.2. - Be prepared to re-bootstrap your buildouts with the the new buildout 2.2 bootstrap script, which will be released soon after the release of distribute 0.7. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton

1 0

Fixing PyPI download stats with real-time log analysis (Was: PyPI Download Counts)
by anatoly techtonik 26 Jun '13

26 Jun '13

On Sun, Jun 9, 2013 at 9:33 PM, Donald Stufft <donald(a)stufft.io> wrote: > > Fastly has been a dream to work with. They've been fast at fixing and > diagnosing issues, have helped tune the config to get a higher hit rate, > and when they heard that people were upset that download counts > had to be turned off they offered the logging support to be turned on > for our account. > > The infrastructure is setup to receive the logs (and intact was receiving > logs for a day or so) but upgrades to the VM that runs PyPI needs to > occur before we can continue receiving them. The disk drive that PyPI > has is to small to handle the volume of request data that is coming in > from Fastly and it quickly filled up in under 24 hours. Upgrading that > space requires powering off the VM so we (The Infra team) are working > on doing that, ideally without downtime on PyPI. > Disk drives? It is unneeded bottleneck. What is needed is resident log cruncher that processes log entries storing intermediate results in memcache, and updates DB counts in batch. This can be even done on AppEngine. Something makes me believe that we can get some extra free quota for this purpose. =) Could you, please, share the log format+example, so that we can experiment with it? -- anatoly t.

4 6

Better way of forcing distutils to use platlib location?
by Yury V. Zaytsev 25 Jun '13

25 Jun '13

Hi, I'm working on Python bindings for a C++ software, which are basically a thin wrapper around a (binary) extension. The extension itself is compiled, linked and installed using autotools. The main motivation for doing it this way stems from the fact that previous attempts to hack distutils classes resulted in a monster script bigger than the autotools build system for the C++ package itself. Still, I'd like to keep using distutils to install the pure Python part. Now that I'm not using the Extension class of distutils to build the extension, however, it doesn't know that the Python stuff should go to platlib along with the extension shared object. So my question is, what is the least horrible way to force distutils to install a pure Python package into platlib instead of purelib? I was thinking of monkeypatching get_python_lib(), but there must certainly be a better solution to the problem that I have overlooked! Thanks and please kindly CC me on replies, -- Sincerely yours, Yury V. Zaytsev

2 3

Re: [Distutils] (no subject)
by Cédric de Saint Martin 25 Jun '13

25 Jun '13

On 24 juin 2013, at 23:24, Tres Seaver <tseaver(a)palladion.com> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 06/24/2013 05:06 PM, Reinout van Rees wrote: > >> Uninstalling distribute? Nice for those with the distribute they got >> from their Ubuntu package... > > Those users need to be lobbying their distro to release a > 'python-setuptools' 0.7.x package to get them out of the dead end they > are in. I'm sorry but... Do we really need the distributions to be updated to have "normal users" able to use buildout? I don't want to start a flame war, but the release/merge of setuptools/distribute gave many users bad headaches. The recent activity on setuptools is really great. The merge was the best thing that could have happened. But honestly, our team already having lost several days of work because of the 0.7 release that didn't work (because of distribute or many different things), when you say "please wait 4 months that a new major release of Ubuntu is out (because I don't think they'll switch so easily to new setuptools in a minor release) so that your users can use buildout", i'm getting mad. I haven't said anything until now, because I know it's *hard* to make this work and be released as expected. But please, take into consideration that the vast majority of setuptools/distribute users don't give a shit about this (most of them don't even know what it is), but they just want something that works without having to loose half a day. Ideally, it should be completely transparent for the user. In the worst case, he should have to do something trivial. I still feel that there is no good reason not to release (for real, i.e on the cheeseshop and so on) a dummy distribute 0.7 depending on setuptools 0.7. Maybe I missed a point? Would it break even more things? Cedric > >> Distribute deprecated and must-be-eradicated? Well, the same was said >> of the virtually unmaintained setuptools. Remember the >> use-pip-plus-distribute soviet-style propaganda images? > > I remember the propaganda, but never bought it. People who *do* buy into > propaganda have some reponsibility for the consequences, no? > >> I don't know why distribute users should bear the pain caused by the >> bad original maintainership of setuptools... > > I think you are missing the point: a distribute 0.7 meta-release is > *for* those users, because it allows them to upgrade from distribute > 0.6.x into the brave new post-merge world. Jason says that there are > some reports that pip cannot upgrade to distribute 0.7.x: I cannot > reproduce that myself. > > > Tres. > - -- > =================================================================== > Tres Seaver +1 540-429-0999 tseaver(a)palladion.com > Palladion Software "Excellence by Design" http://palladion.com > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.11 (GNU/Linux) > Comment: Using GnuPG with undefined - http://www.enigmail.net/ > > iEYEARECAAYFAlHIuQMACgkQ+gerLs4ltQ6q7QCffHjoMMTnnM/D9HFRCtydjWX2 > +IsAn26vmw71lGX/Q8HexN+xnGqntSnx > =6gux > -----END PGP SIGNATURE----- > > -- > You received this message because you are subscribed to the Google Groups "Buildout Development" group. > To unsubscribe from this group and stop receiving emails from it, send an email to buildout-development+unsubscribe(a)googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > >

2 1

Re: [Distutils] Setuptools 0.7.2 and Distribute 0.7 (compatibility wrapper) on PyPI
by Tres Seaver 24 Jun '13

24 Jun '13

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/24/2013 04:44 PM, Jason R. Coombs wrote: > I had to pull distribute 0.7 because older versions of pip couldn't > handle the upgrade. You mean 'pip install -U distribute' fails with distribute 0.7? This works for me in a brand-new virtualenv w/ distribute: $ bin/pip install \ --find-links=https://bitbucket.org/pypa/setuptools/downloads/ \ -U distribute > I'll be re-adding distribute 0.7 back to PyPI following the new > release of pip. Can you test by pointing to the bitbucket downloads in > the short term? > > Can you tell me about what issues the buildout users are experiencing? > Are they trying to upgrade a buildout from distribute to setuptools? > Or is there an issue with creating new buildouts against the latest > setuptools? No issue with the latest setuptools. The "prevent setuptools installation at all costs" behavior of distribute 0.6.x kills the buildout (or its bootstrap) which tries to install setuptools 0.7.2. If there was a clean way to upgrade from distribute 0.6 before running the bootstrap, that would be enough: but then, that was what distribute 0.7 was supposed to do. :( > I'm hesitant to release Distribute 0.7 because even the latest version > will break the 'easy_install' scripts for users who upgrade via pip. > Actually, on second thought, pip users might consider that a feature. pip users can't spell easy_install, so you should be safe. :) > If there's going to be a re-release of distribute 0.7, it'll need to > be coordinated with the pip and virtualenv releases, which we're > planning for the weekend after next (Jul 6). I know that's a long > time, but I'm hoping there's a suitable workaround for buildout. If > not, then getting an updated distribute (and possibly pip) out sooner > (even without immediate) virtualenv support might be possible. Where are the problems pip users reported with distribute 0.7 archived? Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver(a)palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlHItfQACgkQ+gerLs4ltQ5PWQCdHenszGzMEbOXgzmHyIKK/EwW X4oAnRE7uAYmBQE97gabYiWFbjRA1Egz =/k4B -----END PGP SIGNATURE-----

2 1

Setuptools 0.7.2 and Distribute 0.7 (compatibility wrapper) on PyPI
by Jason R. Coombs 24 Jun '13

24 Jun '13

On behalf of the PYPA, I'm excited to announce that Setuptools 0.7 is now official and complete. Released to PyPI, Setuptools 0.7.2 is now available for all to see by default (https://pypi.python.org/pypi/setuptools/). Users of Setuptools 0.6 may upgrade by simply running `easy_install -U setuptools`. Additionally, I released Distribute 0.7 to PyPI. This means that users who have Distribute 0.6.x can upgrade to setuptools 0.7 by simply invoking `easy_install -U distribute` (or `easy_install "distribute>=0.7"`). Windows users should use `easy_install-script -U distribute` (to avoid "file in use" errors on easy_install.exe). The documentation for Setuptools 0.7.2 has been uploaded to https://pythonhosted.org/setuptools and includes the notes about the merge in addition to the official documentation. Please report any issues at the project page (https://bitbucket.org/pypa/setuptools).

8 16

Some more updates to PEP 426
by Nick Coghlan 24 Jun '13

24 Jun '13

Based on the last thread, a few more tweaks to PEP 426: * distlib added as a reference impl in Appendix A * summary field is mandatory again * explicitly advised against using line breaks in the summary and license fields * cleaned up description of the standard setup.py based build system (unified the two separate lists into one, added the missing reference to "build_ext --inplace" that is an implicit precursor to "setup.py test" * explicit advised against invoking "setup.py install" for installation from source (instead recommending building a wheel archive and installing from that - this is what pip will do when caching builds anyway, so we may as well be explicit that if developers want code to run on target systems, they're going to have to adopt metadata 2.0 install hooks) Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Updated drafts of metadata 2.0 (PEP 426 and PEP 440)
by Nick Coghlan 24 Jun '13

24 Jun '13

New versions of PEP 426 and PEP 440 are up on python.org: PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/ PEP 440 (version spec): http://www.python.org/dev/peps/pep-0440/ (as before, not including them inline due to sheer length) The bulk of the changes are in this commit: http://hg.python.org/peps/rev/de305af601fa (there are a few minor tweaks in subsequent commits to the PEPs repo) There have been several significant changes to the main metadata spec in PEP 426: * Added a basic "jsonschema" based description of the format * The "abbreviated metadata" notion is gone, replaced by "included documents". The metadata can now list the names of additional files included in the distinfo directory for the long description, the license and the change history. The contents are no longer embedded directly in the metadata (but will be extracted and made available by PyPI, with the markup format being determined from the file extension, just as it is by sites like GitHub) * The dependency system has been redesigned under the name "Semantic dependencies" (as they now allow a distribution to better say *why* a dependency is needed, rather than just saying "I need this" with almost no capacity to say why). There are now five kinds of dependencies (:meta:, :run:, :test:, :build: and :dev:) and they're integrated into the extras system so you can easily say you want to install some, none or all of them. * The "*" notation is added to extras to make it easier to say "install all dependencies", along with the "-extra_name" notation to exclude the dependencies for specific extras * The "-" notation is added to extras to make it easier to install *just* a distribution's dependencies, without installing the distribution itself. * "Install hooks" are now a distinct concept from the still-hypothetical "metabuild" system, and place more constraints on their expected handling (installation tools are also explicitly permitted to refuse to run them, but doing so is required to fail noisily rather than silently appearing to succeed) The most significant change to PEP 440 is to the handling of pre-releases: whether or not pre-releases should be accepted by default is now determined solely by whether or not there is a stable release that *also* satisfies the version specifier. Reference a pre-release (or not) now has no effect on whether pre-releases are considered viable candidates. Pre-releases are now accepted if: * they're already installed * there's no other available option * the installation tool is told specifically to consider them Other less significant changes to PEP 426 include: * Longer introduction giving more context for the nature and purpose of the metadata * Separated various other things out into appendices * Various tweaks to definitions (including the "Release" tweak from PEP 440, and switching "source archive" to refer to the original raw source, while using "sdist" for the Python specific format with the extra metadata) * Blanket permission for distribution related online services to impose additional restrictions to protect the integrity of the service (such as additional length limits beyond those stated in the PEP). * Explicitly require UTF-8 encoded JSON for serialisation * build_label renamed to source_label * version_url renamed to source_url * tightened up the validation rules for various fields (including RFC references where appropriate). Many of these "new" rules are things projects already comply with (because not complying doesn't work properly). Including them in the spec is about giving PyPI clear guidance to enforce them at point of upload, rather than leaving it to installation tools to try to sort out later. * a few more additions to the "Rejected Features" appendix (notably, my rationale for *not* requiring the install hooks to accept arbitrary keyword arguments) The other PEP 440 changes are also relatively minor: - what were previously called releases are now "final releases", freeing up "release" as a general term for any kind of release (developmental release, pre-release, final release, post release). - "source references" are now "direct references" and can also refer to prebuilt wheel files - automated tools (especially index servers) are given a lot of freedom to be opinionated about the appropriate uses for direct references - a few tweaks to the security guidelines for direct references - pytz/Olson database version translation is explicitly discussed (add a leading 0., convert the trailing letter to an incrementing number) - tighter rules for what characters are allowed in a source label The only major remaining open item is the addition of guidelines in Appendix A for converting legacy metadata to metadata 2.0. I consider the rest of the spec stable at this point, unless anyone picks up on a glaring hole in the latest draft that Daniel, Donald and I missed. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

7 29

[ANN] pypiserver 1.1.2 - minimal private pypi server
by Ralf Schmitt 22 Jun '13

22 Jun '13

Hi, I've just uploaded pypiserver 1.1.2 to the python package index. pypiserver is a minimal PyPI compatible server. It can be used to serve a set of packages and eggs to easy_install or pip. pypiserver is easy to install (i.e. just 'pip install pypiserver'). It doesn't have any external dependencies. https://pypi.python.org/pypi/pypiserver/ should contain enough information to easily get you started running your own PyPI server in a few minutes. The code is available on github: https://github.com/schmir/pypiserver Changes in this version ----------------------- - fix "pypi-server -U" stable/unstable detection, i.e. do not accidentally update to unstable packages -- Cheers Ralf

1 0