[Distutils] User Viewpoint: Packaging History, Confusion and User Advice

Wed Apr 24 17:20:50 CEST 2013

Erik:
Thanks for posting this.
fyi, the "Hitchhikers Guide to Packaging" was recently forked to this:
https://bitbucket.org/pypa/python-packaging-user-guide
This is a pypa project, and and aims to be somewhat authoritative.
It's currently too far out of date to be a good base for meaningful pull
requests.
I'm working locally on it now, and hope to push updates soonish.
At that point, I'd love for people such as yourself, to help make it as
valuable as possible to users.
I'll post here as things develop.
Marcus

On Wed, Apr 24, 2013 at 7:55 AM, Erik Bernoth <erik.bernoth at gmail.com>wrote:

> Hi everybody,
>
> because there is no central place to look at and probably not all
> decisions can be found in written form (like the abandoning of the fellow
> ship list, when and why Tarek Ziadé decided to work on other things then
> distribute/distutils2 and so on) I thought about writing a short summary of
> the current state of packaging efforts as I understand it. I hope it has
> value for people who need to make decisions in packaging their own projects
> as well as help you packaging developers to see it a little bit from your
> user's point of view.
>
> All of it might not be correct or it might be incomplete. It simply
> reflects my current state of investigation about that topic. I also hope
> nobody gets angry, if he sees things differently or considers his effort
> not embraced enough.
>
> Short history about my reasons for this thread
> ---------------------------------------------------------------
>
> End of last year I started a Python project and wanted to make sure users
> can install my project without much effort. The main documentation of
> Python 2.7 confused me deeply and it only talked about distutils and
> setuptools. I simply wanted to add the required other packages, that they
> can be automatically installed with my project, without the user manually
> installing anything (which already let to problems in my project pre-alpha).
>
> For some reason I ran into the pip installer and added, thanks to the
> distutils-sig list [1] , the `install_requires` parameter to the setup()
> function. In conclusion my problem was solved, although I still didn't
> understand anything!
>
> Concluding that being confused might mean that other people are also
> confused, I started an effort to change the distutils part of the core
> documentation [2][3]. I hoped for some guidance, but the basic problem for
> that initiative was that I simply didn't know enough (and probably still
> don't know enough) and nothing could help but investigating. For now I am
> finished with that investigation and will think about continuing about a
> documentation update in the short future.
>
> I am writing this mail, because I hope I can give others and my future
> self an overview of what all the hours of reading half finished documents
> and mailing list threads got me until today. If it makes sense and there is
> a central place to put such information I'd like to improve this text with
> your assistance and put it there.
>
>
> Documentation Advice For Users
> ----------------------------------------------
>
> From all guides to this complicated topic the best one seems to be the
> inofficial guide by Scott Torborg: "How To Package Your Python Code" [4].
> It refuses to discuss all the different projects and their interaction, but
> presents a clear, simple path to follow that gets most things done, at
> least much more then I probably want from packaging in the short future.
>
> A great introduction about the complexities of this topic are in the
> chapter "Python Packaging" in the open book "The Architecture of Open
> Source Applications" [XX]. It's great, because it states different view
> points on the topic, the different requirements and contains diagrams. Oh
> how we people need diagrams to understand complex relationships!
>
> Other resources aren't that useful in their current states: The core
> documentation [5] is confusing, the often cited "Hitchhikers Guide to
> Packaging" [6] seems to be a work in progress from 2009(?), and many
> documentations from packaging tools themselves are not really complete,
> either.
>
> As a side note I like many sections in the distlib documentation [7]. It's
> not complete and has some rough edges, but you can understand a lot about
> what it should do from a first read, without too much frustration. That's
> huge for any tool/framework in it's alpa state! I am a believer that good
> human readable text (a.k.a. a good plan) results in good software, so I
> have high hopes for this development.
>
>
>
> Implementation Advice For Users
> ----------------------------------------------
>
> In most cases setuptools is the way to go. distutils simply doesn't have
> some fundamental features like `install_requires`. Taking other tools
> doesn't seem to be necessary anymore, because distribute should be merged
> into setuptools [8], so under the hood it's features will be used in the
> short future anyway, Distutils2 is not an option due to nobody working on
> it anymore (see the mailing list for example [9]), and distlib is currently
> still in alpha and will likely be added under the hood into setuptools
> anyway, when it's time.
>
>
>
> Packaging efforts/project
> -----------------------------------
>
> (This only consideres projects close to the mainline development. There
> are other tools like bento, or zc.buildout, which might be good or not, but
> don't influence the mainline topics very much, so they are omitted in this
> overview)
>
> distutils (classic) - the old school, deprecated, pretty much unmaintained
> packaging tooling; still the official standard [4] (Note: reasons [13])
>
> setuptools - fork of distutils, which wasn't maintained for a long time
> but will now be merged with it's successor [8]. Contains clear improvements
> over distutils like requirements handling. Not all design decisions are
> still appreciated, while some of them are based on the distutils mistake
> and can't be fixed directly. (finding sources for these arguments will be
> tough, because nobody cites any previous sources, when making such
> arguments, but probably the following 2 threads contain these arguments at
> least once: "Status in packaging 3.3" [10], "packaging location?" [11])
>
> distribute - fork of setuptools [12], which is supported to the current
> day. I don't know if it has any feature improvements over setuptools but I
> guess it should be at least less buggy. Is currently merged back into
> setuptools, which is a work in progress. (Note: Merge notification [8])
>
> distutils2/packaging - a complete rewrite(?) [13] considering new
> requirements, features and bug fixes. Was too big a task and couldn't be
> finished until today. Was planned to be added to Python 3.3, which wasn't
> possible, due to lack of work force for the size of the project. Currently
> abandoned(?)
>
> distlib - following the fail of distutils2 a fundamental set of futures
> shall now be build into a framework that packaging tools like setuptools
> can use to build stable, interoperable packaging mechanisms.
>
> distil - a packaging tool like setuptools purely built on top of the
> current state of distlib (?)
>
> easy_install - an install script that shipped with setuptools.
> maintainance state probably connected to setuptool's (?)
>
> pip - an installer that seems to be developed in parallel with
> distribute(?). From my feeling I'd say there is no much talk about this
> online, because it simply works as expected.
>
> Sources
> --------------------
>
> [1]
> http://mail.python.org/pipermail/distutils-sig/2013-February/019743.html
> [2]
> http://mail.python.org/pipermail/distutils-sig/2013-February/019799.html
> [3] http://mail.python.org/pipermail/docs/2013-March/013550.html
> [4] http://www.scotttorborg.com/python-packaging/minimal.html
> [5] http://docs.python.org/2.7/distutils/index.html (text is the same in
> all following docs' versions as far as I could see)
> [6] http://guide.python-distribute.org/index.html
> [7] http://distlib.readthedocs.org/en/latest/index.html
> [8] http://mail.python.org/pipermail/distutils-sig/2013-March/020126.html
> [9]
> https://groups.google.com/forum/?fromgroups#!forum/the-fellowship-of-the-packaging
> [10]
> http://mail.python.org/pipermail/python-dev/2012-June/thread.html#120430
> [11]
> http://mail.python.org/pipermail/python-dev/2012-September/thread.html#121689
> [12] http://ziade.org/category/packaging,%20python.html
> [13]
> http://ziade.org/2010/03/03/the-fate-of-distutils-pycon-summit-packaging-sprint-detailed-report/
> [XX] http://www.aosabook.org/en/packaging.html (added afterwards as
> honourable mention)
>
> Feedback welcome!
>
> Cheers,
> Erik
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> http://mail.python.org/mailman/listinfo/distutils-sig
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130424/cc01061b/attachment-0001.html>