I would find it helpful to understand the motivation for direct references as
defined in PEP 440, and some clarifications of how they can be used. For
* Can they be used together with other clauses? As PEP 440 appears now, it
seems to indicate that they can, but what does it mean in practice to have a
clause which is so specific with other clauses which are less specific?
* Can we have information in the "Direct reference" section summarising where
such references can and can't appear?
* What direct references provide which cannot be provided by the other version
clauses together with (separately) URIs or paths for where archives can be
found? The fact that the format of this clause is so different from all the
others suggests that it doesn't really belong with them.
There are also questions on other areas:
* It would appear that explicit prefix matching only makes sense for == and !=
clauses (as it is implicit for e.g.<, > clauses). If so, can this be stated
* Since pre-/post-/dev- suffixes are ignored for the purposes of prefix
matching, and given that implicit prefix matching is used with ~= , < and >,
does this mean that explicit prefix matching is only useful for >=, <=?
(given that == with prefix matching appears equivalent to ~=)
It would also be useful to document what kinds of matches would not be possible
without ~= and explicit prefix matching (trailing .*), assuming we had implicit
prefix matching for ==/!= clauses.
Distribute/setuptools 0.7 bring some big changes. These changes can't be
handled by buildout's normal automatic upgrade mechanism. For this reason,
it's important to do one of the following if you're using buildout 2:
- Pin your distribute version to <0.7dev or to some other 0.6 version.
- Upgrade to buildout 2.1.1, which was just released. This will prevent
automatic upgrade to distribute 0.7 or buildout 2.2.
- Be prepared to re-bootstrap your buildouts with the the new buildout 2.2
bootstrap script, which will be released soon after the release of
On Sun, Jun 9, 2013 at 9:33 PM, Donald Stufft <donald(a)stufft.io> wrote:
> Fastly has been a dream to work with. They've been fast at fixing and
> diagnosing issues, have helped tune the config to get a higher hit rate,
> and when they heard that people were upset that download counts
> had to be turned off they offered the logging support to be turned on
> for our account.
> The infrastructure is setup to receive the logs (and intact was receiving
> logs for a day or so) but upgrades to the VM that runs PyPI needs to
> occur before we can continue receiving them. The disk drive that PyPI
> has is to small to handle the volume of request data that is coming in
> from Fastly and it quickly filled up in under 24 hours. Upgrading that
> space requires powering off the VM so we (The Infra team) are working
> on doing that, ideally without downtime on PyPI.
Disk drives? It is unneeded bottleneck. What is needed is resident log
cruncher that processes log entries storing intermediate results in
memcache, and updates DB counts in batch. This can be even done on
AppEngine. Something makes me believe that we can get some extra free quota
for this purpose. =)
Could you, please, share the log format+example, so that we can experiment
I'm working on Python bindings for a C++ software, which are basically a
thin wrapper around a (binary) extension. The extension itself is
compiled, linked and installed using autotools.
The main motivation for doing it this way stems from the fact that
previous attempts to hack distutils classes resulted in a monster script
bigger than the autotools build system for the C++ package itself.
Still, I'd like to keep using distutils to install the pure Python part.
Now that I'm not using the Extension class of distutils to build the
extension, however, it doesn't know that the Python stuff should go to
platlib along with the extension shared object.
So my question is, what is the least horrible way to force distutils to
install a pure Python package into platlib instead of purelib?
I was thinking of monkeypatching get_python_lib(), but there must
certainly be a better solution to the problem that I have overlooked!
Thanks and please kindly CC me on replies,
Yury V. Zaytsev
On 24 juin 2013, at 23:24, Tres Seaver <tseaver(a)palladion.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> On 06/24/2013 05:06 PM, Reinout van Rees wrote:
>> Uninstalling distribute? Nice for those with the distribute they got
>> from their Ubuntu package...
> Those users need to be lobbying their distro to release a
> 'python-setuptools' 0.7.x package to get them out of the dead end they
> are in.
I'm sorry but... Do we really need the distributions to be updated to have "normal users" able to use buildout?
I don't want to start a flame war, but the release/merge of setuptools/distribute gave many users bad headaches.
The recent activity on setuptools is really great. The merge was the best thing that could have happened. But honestly, our team already having lost several days of work because of the 0.7 release that didn't work (because of distribute or many different things), when you say "please wait 4 months that a new major release of Ubuntu is out (because I don't think they'll switch so easily to new setuptools in a minor release) so that your users can use buildout", i'm getting mad.
I haven't said anything until now, because I know it's *hard* to make this work and be released as expected. But please, take into consideration that the vast majority of setuptools/distribute users don't give a shit about this (most of them don't even know what it is), but they just want something that works without having to loose half a day. Ideally, it should be completely transparent for the user. In the worst case, he should have to do something trivial.
I still feel that there is no good reason not to release (for real, i.e on the cheeseshop and so on) a dummy distribute 0.7 depending on setuptools 0.7. Maybe I missed a point? Would it break even more things?
>> Distribute deprecated and must-be-eradicated? Well, the same was said
>> of the virtually unmaintained setuptools. Remember the
>> use-pip-plus-distribute soviet-style propaganda images?
> I remember the propaganda, but never bought it. People who *do* buy into
> propaganda have some reponsibility for the consequences, no?
>> I don't know why distribute users should bear the pain caused by the
>> bad original maintainership of setuptools...
> I think you are missing the point: a distribute 0.7 meta-release is
> *for* those users, because it allows them to upgrade from distribute
> 0.6.x into the brave new post-merge world. Jason says that there are
> some reports that pip cannot upgrade to distribute 0.7.x: I cannot
> reproduce that myself.
> - --
> Tres Seaver +1 540-429-0999 tseaver(a)palladion.com
> Palladion Software "Excellence by Design" http://palladion.com
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.11 (GNU/Linux)
> Comment: Using GnuPG with undefined - http://www.enigmail.net/
> -----END PGP SIGNATURE-----
> You received this message because you are subscribed to the Google Groups "Buildout Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to buildout-development+unsubscribe(a)googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
-----BEGIN PGP SIGNED MESSAGE-----
On 06/24/2013 04:44 PM, Jason R. Coombs wrote:
> I had to pull distribute 0.7 because older versions of pip couldn't
> handle the upgrade.
You mean 'pip install -U distribute' fails with distribute 0.7? This
works for me in a brand-new virtualenv w/ distribute:
$ bin/pip install \
> I'll be re-adding distribute 0.7 back to PyPI following the new
> release of pip. Can you test by pointing to the bitbucket downloads in
> the short term?
> Can you tell me about what issues the buildout users are experiencing?
> Are they trying to upgrade a buildout from distribute to setuptools?
> Or is there an issue with creating new buildouts against the latest
No issue with the latest setuptools. The "prevent setuptools installation
at all costs" behavior of distribute 0.6.x kills the buildout (or its
bootstrap) which tries to install setuptools 0.7.2. If there was a clean
way to upgrade from distribute 0.6 before running the bootstrap, that
would be enough: but then, that was what distribute 0.7 was supposed to
> I'm hesitant to release Distribute 0.7 because even the latest version
> will break the 'easy_install' scripts for users who upgrade via pip.
> Actually, on second thought, pip users might consider that a feature.
pip users can't spell easy_install, so you should be safe. :)
> If there's going to be a re-release of distribute 0.7, it'll need to
> be coordinated with the pip and virtualenv releases, which we're
> planning for the weekend after next (Jul 6). I know that's a long
> time, but I'm hoping there's a suitable workaround for buildout. If
> not, then getting an updated distribute (and possibly pip) out sooner
> (even without immediate) virtualenv support might be possible.
Where are the problems pip users reported with distribute 0.7 archived?
Tres Seaver +1 540-429-0999 tseaver(a)palladion.com
Palladion Software "Excellence by Design" http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/
-----END PGP SIGNATURE-----
On behalf of the PYPA, I'm excited to announce that Setuptools 0.7 is now
official and complete.
Released to PyPI, Setuptools 0.7.2 is now available for all to see by
default (https://pypi.python.org/pypi/setuptools/). Users of Setuptools 0.6
may upgrade by simply running `easy_install -U setuptools`.
Additionally, I released Distribute 0.7 to PyPI. This means that users who
have Distribute 0.6.x can upgrade to setuptools 0.7 by simply invoking
`easy_install -U distribute` (or `easy_install "distribute>=0.7"`). Windows
users should use `easy_install-script -U distribute` (to avoid "file in use"
errors on easy_install.exe).
The documentation for Setuptools 0.7.2 has been uploaded to
https://pythonhosted.org/setuptools and includes the notes about the merge
in addition to the official documentation.
Please report any issues at the project page
Based on the last thread, a few more tweaks to PEP 426:
* distlib added as a reference impl in Appendix A
* summary field is mandatory again
* explicitly advised against using line breaks in the summary and license fields
* cleaned up description of the standard setup.py based build system
(unified the two separate lists into one, added the missing reference
to "build_ext --inplace" that is an implicit precursor to "setup.py
* explicit advised against invoking "setup.py install" for
installation from source (instead recommending building a wheel
archive and installing from that - this is what pip will do when
caching builds anyway, so we may as well be explicit that if
developers want code to run on target systems, they're going to have
to adopt metadata 2.0 install hooks)
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
New versions of PEP 426 and PEP 440 are up on python.org:
PEP 426 (metadata 2.0): http://www.python.org/dev/peps/pep-0426/
PEP 440 (version spec): http://www.python.org/dev/peps/pep-0440/
(as before, not including them inline due to sheer length)
The bulk of the changes are in this commit:
(there are a few minor tweaks in subsequent commits to the PEPs repo)
There have been several significant changes to the main metadata spec
in PEP 426:
* Added a basic "jsonschema" based description of the format
* The "abbreviated metadata" notion is gone, replaced by "included
documents". The metadata can now list the names of additional files
included in the distinfo directory for the long description, the
license and the change history. The contents are no longer embedded
directly in the metadata (but will be extracted and made available by
PyPI, with the markup format being determined from the file extension,
just as it is by sites like GitHub)
* The dependency system has been redesigned under the name "Semantic
dependencies" (as they now allow a distribution to better say *why* a
dependency is needed, rather than just saying "I need this" with
almost no capacity to say why). There are now five kinds of
dependencies (:meta:, :run:, :test:, :build: and :dev:) and they're
integrated into the extras system so you can easily say you want to
install some, none or all of them.
* The "*" notation is added to extras to make it easier to say
"install all dependencies", along with the "-extra_name" notation to
exclude the dependencies for specific extras
* The "-" notation is added to extras to make it easier to install
*just* a distribution's dependencies, without installing the
* "Install hooks" are now a distinct concept from the
still-hypothetical "metabuild" system, and place more constraints on
their expected handling (installation tools are also explicitly
permitted to refuse to run them, but doing so is required to fail
noisily rather than silently appearing to succeed)
The most significant change to PEP 440 is to the handling of
pre-releases: whether or not pre-releases should be accepted by
default is now determined solely by whether or not there is a stable
release that *also* satisfies the version specifier. Reference a
pre-release (or not) now has no effect on whether pre-releases are
considered viable candidates. Pre-releases are now accepted if:
* they're already installed
* there's no other available option
* the installation tool is told specifically to consider them
Other less significant changes to PEP 426 include:
* Longer introduction giving more context for the nature and purpose
of the metadata
* Separated various other things out into appendices
* Various tweaks to definitions (including the "Release" tweak from
PEP 440, and switching "source archive" to refer to the original raw
source, while using "sdist" for the Python specific format with the
* Blanket permission for distribution related online services to
impose additional restrictions to protect the integrity of the service
(such as additional length limits beyond those stated in the PEP).
* Explicitly require UTF-8 encoded JSON for serialisation
* build_label renamed to source_label
* version_url renamed to source_url
* tightened up the validation rules for various fields (including RFC
references where appropriate). Many of these "new" rules are things
projects already comply with (because not complying doesn't work
properly). Including them in the spec is about giving PyPI clear
guidance to enforce them at point of upload, rather than leaving it to
installation tools to try to sort out later.
* a few more additions to the "Rejected Features" appendix (notably,
my rationale for *not* requiring the install hooks to accept arbitrary
The other PEP 440 changes are also relatively minor:
- what were previously called releases are now "final releases",
freeing up "release" as a general term for any kind of release
(developmental release, pre-release, final release, post release).
- "source references" are now "direct references" and can also refer
to prebuilt wheel files
- automated tools (especially index servers) are given a lot of
freedom to be opinionated about the appropriate uses for direct
- a few tweaks to the security guidelines for direct references
- pytz/Olson database version translation is explicitly discussed (add
a leading 0., convert the trailing letter to an incrementing number)
- tighter rules for what characters are allowed in a source label
The only major remaining open item is the addition of guidelines in
Appendix A for converting legacy metadata to metadata 2.0. I consider
the rest of the spec stable at this point, unless anyone picks up on a
glaring hole in the latest draft that Daniel, Donald and I missed.
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
I've just uploaded pypiserver 1.1.2 to the python package index.
pypiserver is a minimal PyPI compatible server. It can be used to serve
a set of packages and eggs to easy_install or pip.
pypiserver is easy to install (i.e. just 'pip install pypiserver'). It
doesn't have any external dependencies.
https://pypi.python.org/pypi/pypiserver/ should contain enough
information to easily get you started running your own PyPI server in a
The code is available on github: https://github.com/schmir/pypiserver
Changes in this version
- fix "pypi-server -U" stable/unstable detection, i.e. do not
accidentally update to unstable packages