On Thu, Apr 25, 2013 at 4:58 PM, Daniel Holth <dholth(a)gmail.com> wrote:
> I would prefer to see PEP 390 withdrawn and I think this has been
> suggested before. The metadata is already sourced from different files
> depending on your build system.
> The .ini format and our parsers for it are really awful. I always
> resented having to learn it in order to use distutils/setuptools when
> every other language (RFC822, Python, JSON) is both better and already
Everyone here has an excellent user stories. I am stunned that there is not
process to help PEP authors collect them? Original user stories are much
better to learn and draft new standards than when reworked into PEP
> FWIW bdist_wheel does something half-PEP-390 inspired with setup.cfg:
> provides-extra =
> requires-dist =
> distribute (>= 0.6.34)
> argparse; python_version == '2.6'
> keyring; extra == 'signatures'
> dirspec; sys.platform != 'win32' and extra == 'signatures'
> ed25519ll; extra == 'faster-signatures'
> license-file = LICENSE.txt
Why reinvent own format when there is already YAML with indented sections?
Yes, I have to see examples to write in this format, but it is readable and
familiar across a broad range of products. It is better than JSON at least,
because it doesn't require to wrap anything in quotes.
I was just fiddling with an experimental environment marker
implementation for setuptools, and ran across a bit of a quirk in the
spec. It says that the value of 'extra' is 'None', but that
comparisons can only be done on strings.
This leads to two problems: first, the syntax doesn't have a way to
spell 'None', so you can't actually compare it to something. Second,
if you compare it to a string using the equality operators, IIUC
you'll get a TypeError under Python 3.
Should it actually be defaulting 'extra' to an empty string? That
would actually make a lot more sense, and avoid the issue of
implementing non-string comparisons, None literals, etc.
(Doing extras in this way actually has another problem, btw, which is
that it's insane to do a != comparison on an 'extra'. And there are
probably other insane operations on extras, because unlike everything
else, the extra would need to be dynamically evaluated. I think it
would probably be an improvement to remove extras from the environment
marker system altogether and just have a mapping of extras instead,
ala setuptools. But that can be treated as a separate issue from the
Having a lot of meaningless options doesn't make meta data any more clear.
Well, they are not meaningless, but their role is fully fulfilled by other
options (Author and Maintainer fields in this particular case).
The use case for the Author field is that if Author wants to be contacted,
(s)he leaves email. That's it. This use case should be described in the
meta-data along with the format that are expected to be recognized by the
Author: anatoly techtonik
Author: anatoly techtonik <techtonik(a)gmail.com>
Author: anatoly techtonik <techtonik(a)gmail.com>, Anything Else for Humans,
Or For Future Specs
Here the field content defines its type - it's like duck typing for
specification, which make specifications more pythonic.
Is it good?
because there is no central place to look at and probably not all decisions
can be found in written form (like the abandoning of the fellow ship list,
when and why Tarek Ziadé decided to work on other things then
distribute/distutils2 and so on) I thought about writing a short summary of
the current state of packaging efforts as I understand it. I hope it has
value for people who need to make decisions in packaging their own projects
as well as help you packaging developers to see it a little bit from your
user's point of view.
All of it might not be correct or it might be incomplete. It simply
reflects my current state of investigation about that topic. I also hope
nobody gets angry, if he sees things differently or considers his effort
not embraced enough.
Short history about my reasons for this thread
End of last year I started a Python project and wanted to make sure users
can install my project without much effort. The main documentation of
Python 2.7 confused me deeply and it only talked about distutils and
setuptools. I simply wanted to add the required other packages, that they
can be automatically installed with my project, without the user manually
installing anything (which already let to problems in my project pre-alpha).
For some reason I ran into the pip installer and added, thanks to the
distutils-sig list  , the `install_requires` parameter to the setup()
function. In conclusion my problem was solved, although I still didn't
Concluding that being confused might mean that other people are also
confused, I started an effort to change the distutils part of the core
documentation . I hoped for some guidance, but the basic problem for
that initiative was that I simply didn't know enough (and probably still
don't know enough) and nothing could help but investigating. For now I am
finished with that investigation and will think about continuing about a
documentation update in the short future.
I am writing this mail, because I hope I can give others and my future self
an overview of what all the hours of reading half finished documents and
mailing list threads got me until today. If it makes sense and there is a
central place to put such information I'd like to improve this text with
your assistance and put it there.
Documentation Advice For Users
>From all guides to this complicated topic the best one seems to be the
inofficial guide by Scott Torborg: "How To Package Your Python Code" .
It refuses to discuss all the different projects and their interaction, but
presents a clear, simple path to follow that gets most things done, at
least much more then I probably want from packaging in the short future.
A great introduction about the complexities of this topic are in the
chapter "Python Packaging" in the open book "The Architecture of Open
Source Applications" [XX]. It's great, because it states different view
points on the topic, the different requirements and contains diagrams. Oh
how we people need diagrams to understand complex relationships!
Other resources aren't that useful in their current states: The core
documentation  is confusing, the often cited "Hitchhikers Guide to
Packaging"  seems to be a work in progress from 2009(?), and many
documentations from packaging tools themselves are not really complete,
As a side note I like many sections in the distlib documentation . It's
not complete and has some rough edges, but you can understand a lot about
what it should do from a first read, without too much frustration. That's
huge for any tool/framework in it's alpa state! I am a believer that good
human readable text (a.k.a. a good plan) results in good software, so I
have high hopes for this development.
Implementation Advice For Users
In most cases setuptools is the way to go. distutils simply doesn't have
some fundamental features like `install_requires`. Taking other tools
doesn't seem to be necessary anymore, because distribute should be merged
into setuptools , so under the hood it's features will be used in the
short future anyway, Distutils2 is not an option due to nobody working on
it anymore (see the mailing list for example ), and distlib is currently
still in alpha and will likely be added under the hood into setuptools
anyway, when it's time.
(This only consideres projects close to the mainline development. There are
other tools like bento, or zc.buildout, which might be good or not, but
don't influence the mainline topics very much, so they are omitted in this
distutils (classic) - the old school, deprecated, pretty much unmaintained
packaging tooling; still the official standard  (Note: reasons )
setuptools - fork of distutils, which wasn't maintained for a long time but
will now be merged with it's successor . Contains clear improvements
over distutils like requirements handling. Not all design decisions are
still appreciated, while some of them are based on the distutils mistake
and can't be fixed directly. (finding sources for these arguments will be
tough, because nobody cites any previous sources, when making such
arguments, but probably the following 2 threads contain these arguments at
least once: "Status in packaging 3.3" , "packaging location?" )
distribute - fork of setuptools , which is supported to the current
day. I don't know if it has any feature improvements over setuptools but I
guess it should be at least less buggy. Is currently merged back into
setuptools, which is a work in progress. (Note: Merge notification )
distutils2/packaging - a complete rewrite(?)  considering new
requirements, features and bug fixes. Was too big a task and couldn't be
finished until today. Was planned to be added to Python 3.3, which wasn't
possible, due to lack of work force for the size of the project. Currently
distlib - following the fail of distutils2 a fundamental set of futures
shall now be build into a framework that packaging tools like setuptools
can use to build stable, interoperable packaging mechanisms.
distil - a packaging tool like setuptools purely built on top of the
current state of distlib (?)
easy_install - an install script that shipped with setuptools. maintainance
state probably connected to setuptool's (?)
pip - an installer that seems to be developed in parallel with
distribute(?). From my feeling I'd say there is no much talk about this
online, because it simply works as expected.
 http://docs.python.org/2.7/distutils/index.html (text is the same in
all following docs' versions as far as I could see)
[XX] http://www.aosabook.org/en/packaging.html (added afterwards as
fyi, there is dev list now for the "pypa" family of projects (most notably
pip and virtualenv)
This is *not* meant to replace distutils-sig.
It's for more "day-to-day" discussions related to maintaining and releasing
the "pypa" projects.
I've made progress in my experiment to replace pkg_resources usage in pip
with code in distlib. The basic approach is to replace references to
pkg_resources with references to pip.vendor.distlib.pkg_resources, where the
latter module is a shim to emulate the pkg_resources API using distlib to
actually implement the functionality.
Although I don't have the very latest upstream pip updates merged in yet, I
have test results which mirror the develop branch (one failure, related to my
test environment - test_correct_pip_version). I believe I have now replaced
all uses of pkg_resources (when I mentioned this effort previously on this
list, pip/req.py had not yet been converted).
Note that the pip.vendor.distlib version is slightly different to the
released distlib - the main difference being that the metadata version in the
released distlib is treated as 2.0, whereas for pip I had to change it to
The code is at  - the use-distlib branch - and a comparison view is at
I would appreciate comments, especially from the pip maintainers.
The PEP 426 meta-data format allows only single Author and single
Maintainer per package, which is very inadequate to the way most popular
packages are developed.
The format should contains recommendation how to handle credits and process
With the proper implementation the packaging format can positively affect
social collaboration and bring more fun into development. I'd be interested
to know the opinion if somebody considers bringing this topic outside of
the distutils sig.
I have a Python extension which uses CPU-specific features,
if available. This is done through a run-time check. If the
hardware supports the POPCNT instruction then it selects one
implementation of my inner loop, if SSSE3 is available then
it selects another, otherwise it falls back to generic versions
of my performance critical kernel. (Some 95%+ of the time is
spent in this kernel.)
Unfortunately, there's a failure mode I didn't expect. I
use -mssse3 and -O3 to compile all of the C code, even though
only one file needs that -mssse3 option.
As a result, the other files are compiled with the expectation
that SSSE3 will exist. This causes a segfault for the line
start_target_popcount = (int)(query_popcount * threshold);
because the compiler used fisttpl, which is an SSSE-3 instruction.
After all, I told it to assume that ssse3 exists.
The Debian packager for my package recently ran into this problem,
because the test machine has a gcc which understands -mssse3 but
the machine itself has an older CPU without those instructions.
I'm trying to come up with a solution that can be automated for
the Debian distribution. I want a solution where the same binary
can work on older machines and on newer ones
Ideally I would like to say that only one file is compiled
with the -mssse3 option. Since my selector code isn't part of this
file, SSSE-3 code will never be executed unless the CPU supports is.
However, I can't figure out any way to tell distutils that
a set of compiler options are specific to a single file.
Is that even possible?
Since the password reset on PyPI site (https://pypi.python.org/pypi), I
cannot longer log in there. My username used to 'falted' there, and
although I requested a reset password for this user
the form says that a mail has been sent, the thing is that I don't
receive any e-mail. I am not sure why I'm not receiving this message!
I need to continue providing releases for the numexpr, blosc and other
packages, but I can't. What can I do for gaining access to my 'falted'
On IRC yesterday, Spanktar asked a question that comes up every so
"How do I disable logins while I do some maintenance on my Plone site?"
Various suggests were offered, including davisagli's (paraphrasing):
"I usually disable all the PAS auth plugins in acl_users"
DING DING DING: LIGHT BULB MOMENT. I've seen this question asked enough
times to think it might be nice if there were a checkbox in Site Setup
-> Users & Groups to make it easy for end users to perform this action,
if only someone were willing to do the work…
Alex Clark · http://about.me/alex.clark