Mailman 3 April 2013 - Distutils-SIG

.ini vs JSON vs YAML meta data format (Was: Simplify 426: Deprecate Author-email and Maintainer-email)
by anatoly techtonik 28 Apr '13

28 Apr '13

On Thu, Apr 25, 2013 at 4:58 PM, Daniel Holth <dholth(a)gmail.com> wrote: > I would prefer to see PEP 390 withdrawn and I think this has been > suggested before. The metadata is already sourced from different files > depending on your build system. > > The .ini format and our parsers for it are really awful. I always > resented having to learn it in order to use distutils/setuptools when > every other language (RFC822, Python, JSON) is both better and already > familiar. > Everyone here has an excellent user stories. I am stunned that there is not process to help PEP authors collect them? Original user stories are much better to learn and draft new standards than when reworked into PEP specification text. > FWIW bdist_wheel does something half-PEP-390 inspired with setup.cfg: > > [metadata] > provides-extra = > tool > signatures > faster-signatures > requires-dist = > distribute (>= 0.6.34) > argparse; python_version == '2.6' > keyring; extra == 'signatures' > dirspec; sys.platform != 'win32' and extra == 'signatures' > ed25519ll; extra == 'faster-signatures' > license-file = LICENSE.txt > > (https://bitbucket.org/dholth/wheel/src/tip/setup.cfg?at=default) Why reinvent own format when there is already YAML with indented sections? Yes, I have to see examples to write in this format, but it is readable and familiar across a broad range of products. It is better than JSON at least, because it doesn't require to wrap anything in quotes.

4 6

Environment marker expression types
by PJ Eby 28 Apr '13

28 Apr '13

I was just fiddling with an experimental environment marker implementation for setuptools, and ran across a bit of a quirk in the spec. It says that the value of 'extra' is 'None', but that comparisons can only be done on strings. This leads to two problems: first, the syntax doesn't have a way to spell 'None', so you can't actually compare it to something. Second, if you compare it to a string using the equality operators, IIUC you'll get a TypeError under Python 3. Should it actually be defaulting 'extra' to an empty string? That would actually make a lot more sense, and avoid the issue of implementing non-string comparisons, None literals, etc. (Doing extras in this way actually has another problem, btw, which is that it's insane to do a != comparison on an 'extra'. And there are probably other insane operations on extras, because unlike everything else, the extra would need to be dynamically evaluated. I think it would probably be an improvement to remove extras from the environment marker system altogether and just have a mapping of extras instead, ala setuptools. But that can be treated as a separate issue from the 'None' problem.)

5 17

Simplify 426: Deprecate Author-email and Maintainer-email
by anatoly techtonik 27 Apr '13

27 Apr '13

Having a lot of meaningless options doesn't make meta data any more clear. Well, they are not meaningless, but their role is fully fulfilled by other options (Author and Maintainer fields in this particular case). The use case for the Author field is that if Author wants to be contacted, (s)he leaves email. That's it. This use case should be described in the meta-data along with the format that are expected to be recognized by the software: Author: anatoly techtonik Author: anatoly techtonik <techtonik(a)gmail.com> Author: anatoly techtonik <techtonik(a)gmail.com>, Anything Else for Humans, Or For Future Specs Here the field content defines its type - it's like duck typing for specification, which make specifications more pythonic. Is it good? -- anatoly t.

8 18

User Viewpoint: Packaging History, Confusion and User Advice
by Erik Bernoth 26 Apr '13

26 Apr '13

Hi everybody, because there is no central place to look at and probably not all decisions can be found in written form (like the abandoning of the fellow ship list, when and why Tarek Ziadé decided to work on other things then distribute/distutils2 and so on) I thought about writing a short summary of the current state of packaging efforts as I understand it. I hope it has value for people who need to make decisions in packaging their own projects as well as help you packaging developers to see it a little bit from your user's point of view. All of it might not be correct or it might be incomplete. It simply reflects my current state of investigation about that topic. I also hope nobody gets angry, if he sees things differently or considers his effort not embraced enough. Short history about my reasons for this thread --------------------------------------------------------------- End of last year I started a Python project and wanted to make sure users can install my project without much effort. The main documentation of Python 2.7 confused me deeply and it only talked about distutils and setuptools. I simply wanted to add the required other packages, that they can be automatically installed with my project, without the user manually installing anything (which already let to problems in my project pre-alpha). For some reason I ran into the pip installer and added, thanks to the distutils-sig list [1] , the `install_requires` parameter to the setup() function. In conclusion my problem was solved, although I still didn't understand anything! Concluding that being confused might mean that other people are also confused, I started an effort to change the distutils part of the core documentation [2][3]. I hoped for some guidance, but the basic problem for that initiative was that I simply didn't know enough (and probably still don't know enough) and nothing could help but investigating. For now I am finished with that investigation and will think about continuing about a documentation update in the short future. I am writing this mail, because I hope I can give others and my future self an overview of what all the hours of reading half finished documents and mailing list threads got me until today. If it makes sense and there is a central place to put such information I'd like to improve this text with your assistance and put it there. Documentation Advice For Users ---------------------------------------------- >From all guides to this complicated topic the best one seems to be the inofficial guide by Scott Torborg: "How To Package Your Python Code" [4]. It refuses to discuss all the different projects and their interaction, but presents a clear, simple path to follow that gets most things done, at least much more then I probably want from packaging in the short future. A great introduction about the complexities of this topic are in the chapter "Python Packaging" in the open book "The Architecture of Open Source Applications" [XX]. It's great, because it states different view points on the topic, the different requirements and contains diagrams. Oh how we people need diagrams to understand complex relationships! Other resources aren't that useful in their current states: The core documentation [5] is confusing, the often cited "Hitchhikers Guide to Packaging" [6] seems to be a work in progress from 2009(?), and many documentations from packaging tools themselves are not really complete, either. As a side note I like many sections in the distlib documentation [7]. It's not complete and has some rough edges, but you can understand a lot about what it should do from a first read, without too much frustration. That's huge for any tool/framework in it's alpa state! I am a believer that good human readable text (a.k.a. a good plan) results in good software, so I have high hopes for this development. Implementation Advice For Users ---------------------------------------------- In most cases setuptools is the way to go. distutils simply doesn't have some fundamental features like `install_requires`. Taking other tools doesn't seem to be necessary anymore, because distribute should be merged into setuptools [8], so under the hood it's features will be used in the short future anyway, Distutils2 is not an option due to nobody working on it anymore (see the mailing list for example [9]), and distlib is currently still in alpha and will likely be added under the hood into setuptools anyway, when it's time. Packaging efforts/project ----------------------------------- (This only consideres projects close to the mainline development. There are other tools like bento, or zc.buildout, which might be good or not, but don't influence the mainline topics very much, so they are omitted in this overview) distutils (classic) - the old school, deprecated, pretty much unmaintained packaging tooling; still the official standard [4] (Note: reasons [13]) setuptools - fork of distutils, which wasn't maintained for a long time but will now be merged with it's successor [8]. Contains clear improvements over distutils like requirements handling. Not all design decisions are still appreciated, while some of them are based on the distutils mistake and can't be fixed directly. (finding sources for these arguments will be tough, because nobody cites any previous sources, when making such arguments, but probably the following 2 threads contain these arguments at least once: "Status in packaging 3.3" [10], "packaging location?" [11]) distribute - fork of setuptools [12], which is supported to the current day. I don't know if it has any feature improvements over setuptools but I guess it should be at least less buggy. Is currently merged back into setuptools, which is a work in progress. (Note: Merge notification [8]) distutils2/packaging - a complete rewrite(?) [13] considering new requirements, features and bug fixes. Was too big a task and couldn't be finished until today. Was planned to be added to Python 3.3, which wasn't possible, due to lack of work force for the size of the project. Currently abandoned(?) distlib - following the fail of distutils2 a fundamental set of futures shall now be build into a framework that packaging tools like setuptools can use to build stable, interoperable packaging mechanisms. distil - a packaging tool like setuptools purely built on top of the current state of distlib (?) easy_install - an install script that shipped with setuptools. maintainance state probably connected to setuptool's (?) pip - an installer that seems to be developed in parallel with distribute(?). From my feeling I'd say there is no much talk about this online, because it simply works as expected. Sources -------------------- [1] http://mail.python.org/pipermail/distutils-sig/2013-February/019743.html [2] http://mail.python.org/pipermail/distutils-sig/2013-February/019799.html [3] http://mail.python.org/pipermail/docs/2013-March/013550.html [4] http://www.scotttorborg.com/python-packaging/minimal.html [5] http://docs.python.org/2.7/distutils/index.html (text is the same in all following docs' versions as far as I could see) [6] http://guide.python-distribute.org/index.html [7] http://distlib.readthedocs.org/en/latest/index.html [8] http://mail.python.org/pipermail/distutils-sig/2013-March/020126.html [9] https://groups.google.com/forum/?fromgroups#!forum/the-fellowship-of-the-pa… [10] http://mail.python.org/pipermail/python-dev/2012-June/thread.html#120430 [11] http://mail.python.org/pipermail/python-dev/2012-September/thread.html#1216… [12] http://ziade.org/category/packaging,%20python.html [13] http://ziade.org/2010/03/03/the-fate-of-distutils-pycon-summit-packaging-sp… [XX] http://www.aosabook.org/en/packaging.html (added afterwards as honourable mention) Feedback welcome! Cheers, Erik

6 10

pypa-dev mailing list
by Marcus Smith 24 Apr '13

24 Apr '13

fyi, there is dev list now for the "pypa" family of projects (most notably pip and virtualenv) " "pypa-dev" https://groups.google.com/forum/?fromgroups#!forum/pypa-dev This is *not* meant to replace distutils-sig. It's for more "day-to-day" discussions related to maintaining and releasing the "pypa" projects. Marcus

4 6

Replacing pkg_resources with distlib in pip
by Vinay Sajip 24 Apr '13

24 Apr '13

I've made progress in my experiment to replace pkg_resources usage in pip with code in distlib. The basic approach is to replace references to pkg_resources with references to pip.vendor.distlib.pkg_resources, where the latter module is a shim to emulate the pkg_resources API using distlib to actually implement the functionality. Although I don't have the very latest upstream pip updates merged in yet, I have test results which mirror the develop branch (one failure, related to my test environment - test_correct_pip_version). I believe I have now replaced all uses of pkg_resources (when I mentioned this effort previously on this list, pip/req.py had not yet been converted). Note that the pip.vendor.distlib version is slightly different to the released distlib - the main difference being that the metadata version in the released distlib is treated as 2.0, whereas for pip I had to change it to 1.3. The code is at [1] - the use-distlib branch - and a comparison view is at [2]. I would appreciate comments, especially from the pip maintainers. Regards, Vinay Sajip [1] https://github.com/vsajip/pip [2] https://github.com/vsajip/pip/compare/use-distlib

6 7

Enhance 426: Insufficient crediting options
by anatoly techtonik 24 Apr '13

24 Apr '13

The PEP 426 meta-data format allows only single Author and single Maintainer per package, which is very inadequate to the way most popular packages are developed. The format should contains recommendation how to handle credits and process them. With the proper implementation the packaging format can positively affect social collaboration and bring more fun into development. I'd be interested to know the opinion if somebody considers bringing this topic outside of the distutils sig. -- anatoly t.

1 0

special compiler options for only one file
by Andrew Dalke 23 Apr '13

23 Apr '13

Hi all, I have a Python extension which uses CPU-specific features, if available. This is done through a run-time check. If the hardware supports the POPCNT instruction then it selects one implementation of my inner loop, if SSSE3 is available then it selects another, otherwise it falls back to generic versions of my performance critical kernel. (Some 95%+ of the time is spent in this kernel.) Unfortunately, there's a failure mode I didn't expect. I use -mssse3 and -O3 to compile all of the C code, even though only one file needs that -mssse3 option. As a result, the other files are compiled with the expectation that SSSE3 will exist. This causes a segfault for the line start_target_popcount = (int)(query_popcount * threshold); because the compiler used fisttpl, which is an SSSE-3 instruction. After all, I told it to assume that ssse3 exists. The Debian packager for my package recently ran into this problem, because the test machine has a gcc which understands -mssse3 but the machine itself has an older CPU without those instructions. I'm trying to come up with a solution that can be automated for the Debian distribution. I want a solution where the same binary can work on older machines and on newer ones Ideally I would like to say that only one file is compiled with the -mssse3 option. Since my selector code isn't part of this file, SSSE-3 code will never be executed unless the CPU supports is. However, I can't figure out any way to tell distutils that a set of compiler options are specific to a single file. Is that even possible? Cheers, Andrew dalke(a)dalkescientific.com

3 3

Problems with PyPI authentication
by Francesc Alted 21 Apr '13

21 Apr '13

Hi, Since the password reset on PyPI site (https://pypi.python.org/pypi), I cannot longer log in there. My username used to 'falted' there, and although I requested a reset password for this user (https://pypi.python.org/pypi?%3Aaction=forgotten_password_form), and the form says that a mail has been sent, the thing is that I don't receive any e-mail. I am not sure why I'm not receiving this message! I need to continue providing releases for the numexpr, blosc and other packages, but I can't. What can I do for gaining access to my 'falted' account again? Thanks, -- Francesc Alted

1 0

Free GSOC idea: Site Setup -> Users & Groups -> [ ] Disable logins
by Alex Clark 20 Apr '13

20 Apr '13

On IRC yesterday, Spanktar asked a question that comes up every so often (paraphrasing): "How do I disable logins while I do some maintenance on my Plone site?" Various suggests were offered, including davisagli's (paraphrasing): "I usually disable all the PAS auth plugins in acl_users" DING DING DING: LIGHT BULB MOMENT. I've seen this question asked enough times to think it might be nice if there were a checkbox in Site Setup -> Users & Groups to make it easy for end users to perform this action, if only someone were willing to do the work… -- Alex Clark · http://about.me/alex.clark

1 1