advice re: packaging tasks
Hi, I was curious what others do for the following packaging tasks, or if you have any recommendations otherwise. There is also a code organization question at the end. 1) For starters, it's very easy to make mistakes in one's MANIFEST.in, so I hacked the sdist command in my setup.py to list the differences between one's project repo and the generated sdist each time you run "python setup.py sdist". Are there better solutions for this out there so I don't have to rely on my own hack? 2) Secondly, like many, my README files are in markdown, so I hacked a command in my setup.py to use Pandoc to convert README.md to a .rst file for use as the long_description argument to setup(). I also check in the resulting file for troubleshooting purposes, etc. Are there more elegant solutions for this that people know of? Also, for commands like the latter, is it better to define them in one's setup.py, or simply to have separate scripts in one's repo? Lastly, as these setup-related tasks grow larger and more complicated, I found it helped to break them out into a separate setup package that sits alongside my project's main package library (and even adding tests in some cases). Is this normal? Have other people run into this? Thanks, --Chris
On Sep 30, 2014, at 7:44 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
Hi,
I was curious what others do for the following packaging tasks, or if you have any recommendations otherwise. There is also a code organization question at the end.
1) For starters, it's very easy to make mistakes in one's MANIFEST.in, so I hacked the sdist command in my setup.py to list the differences between one's project repo and the generated sdist each time you run "python setup.py sdist". Are there better solutions for this out there so I don't have to rely on my own hack?
https://warehouse.python.org/project/check-manifest/
2) Secondly, like many, my README files are in markdown, so I hacked a command in my setup.py to use Pandoc to convert README.md to a .rst file for use as the long_description argument to setup(). I also check in the resulting file for troubleshooting purposes, etc. Are there more elegant solutions for this that people know of?
Not that I’m aware of.
Also, for commands like the latter, is it better to define them in one's setup.py, or simply to have separate scripts in one's repo?
I’d probably do it in the setup.py, or as an invoke task, see https://warehouse.python.org/project/invoke/.
Lastly, as these setup-related tasks grow larger and more complicated, I found it helped to break them out into a separate setup package that sits alongside my project's main package library (and even adding tests in some cases). Is this normal? Have other people run into this?
This is especially the case when I start using invoke.
Thanks, --Chris _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Hi Chris, On 09/30/2014 05:44 PM, Chris Jerdonek wrote:
2) Secondly, like many, my README files are in markdown, so I hacked a command in my setup.py to use Pandoc to convert README.md to a .rst file for use as the long_description argument to setup(). I also check in the resulting file for troubleshooting purposes, etc. Are there more elegant solutions for this that people know of?
Perhaps this is stating the obvious, but the simplest solution here is just to switch your README to rst format. It's not that different from Markdown, GitHub/BitBucket will parse it and display it nicely, etc. Carl
On Tue, Sep 30, 2014 at 04:44:09PM -0700, Chris Jerdonek wrote:
I was curious what others do for the following packaging tasks, or if you have any recommendations otherwise. There is also a code organization question at the end.
1) For starters, it's very easy to make mistakes in one's MANIFEST.in, so I hacked the sdist command in my setup.py to list the differences between one's project repo and the generated sdist each time you run "python setup.py sdist". Are there better solutions for this out there so I don't have to rely on my own hack?
I wrote check-manifest for this. Prior to that I used to have horrific Makefile rules that did a similar check: https://github.com/mgedmin/objgraph/blob/master/Makefile#L69
2) Secondly, like many, my README files are in markdown, so I hacked a command in my setup.py to use Pandoc to convert README.md to a .rst file for use as the long_description argument to setup(). I also check in the resulting file for troubleshooting purposes, etc. Are there more elegant solutions for this that people know of?
I wrote restview for a different purpose, but found it rather useful for discovering ReStructuredText problems that would make PyPI's fall back to plaintext rendering. If you run restview --long-description in a directory containing a setup.py, it'll invoke `python setup.py --long-description' and interpret it using the same docutils settings as PyPI, highlighting any errors. For a more automated (but perhaps less accurate) solution I pipe python setup.py --long-description | rst2html > /dev/null in my Makefile: https://github.com/mgedmin/objgraph/blob/master/Makefile#L99
Also, for commands like the latter, is it better to define them in one's setup.py, or simply to have separate scripts in one's repo?
I'm really happy with being able to make releases of any of my packages with make release This runs the test suite (tox/detox FTW), checks for uncommitted changes, makes sure MANIFEST.in is correct, makes sure the version number in setup.py matches the version number in CHANGES.rst (and release date is today), makes sure there are no RST errors in long_descrption, and reminds me the commands I need to run to creates a version control system tag and package + upload the sdist to PyPI. (I'm too paranoid to let automated tools perform externally-visible actions like uploading to PyPI or pushing git tags, so I just copy & paste the commands once I'm sure everything's ship-shape.) There are other excellent tools for this (e.g. zest.releaser). One day I'll get tired of copy-pasting my Makefiles from project to project and I'll switch to zest.releaser.
Lastly, as these setup-related tasks grow larger and more complicated, I found it helped to break them out into a separate setup package that sits alongside my project's main package library (and even adding tests in some cases). Is this normal? Have other people run into this?
I'm not sure what you mean. Do you have any examples? Marius Gedminas -- Life begins when you can spend your spare time programming instead of watching television. -- Cal Keegan
On Thu, Oct 2, 2014 at 8:08 AM, Marius Gedminas <marius@pov.lt> wrote:
On Tue, Sep 30, 2014 at 04:44:09PM -0700, Chris Jerdonek wrote:
I was curious what others do for the following packaging tasks, or if you have any recommendations otherwise. There is also a code organization question at the end.
1) For starters, it's very easy to make mistakes in one's MANIFEST.in, so I hacked the sdist command in my setup.py to list the differences between one's project repo and the generated sdist each time you run "python setup.py sdist". Are there better solutions for this out there so I don't have to rely on my own hack?
I wrote check-manifest for this.
Yes, I just started using this (as you know). :) It is very nice!
Prior to that I used to have horrific Makefile rules that did a similar check: https://github.com/mgedmin/objgraph/blob/master/Makefile#L69
2) Secondly, like many, my README files are in markdown, so I hacked a command in my setup.py to use Pandoc to convert README.md to a .rst file for use as the long_description argument to setup(). I also check in the resulting file for troubleshooting purposes, etc. Are there more elegant solutions for this that people know of?
I wrote restview for a different purpose, but found it rather useful for discovering ReStructuredText problems that would make PyPI's fall back to plaintext rendering.
If you run
restview --long-description
in a directory containing a setup.py, it'll invoke `python setup.py --long-description' and interpret it using the same docutils settings as PyPI, highlighting any errors.
For a more automated (but perhaps less accurate) solution I pipe
python setup.py --long-description | rst2html > /dev/null
Yes, I had been familiar with this latter approach. It is documented here: https://docs.python.org/2/distutils/packageindex.html#pypi-package-display It did not work for me in my case though. I eventually found that my use of URL fragments to link to sections elsewhere on the same page, and using relative links (supported by GitHub, https://github.com/blog/1395-relative-links-in-markup-files ) are what broke things. Here is a link to a PyPI bug report regarding making it easier to find such issues: https://bitbucket.org/pypa/pypi/issue/161/rest-formatting-fails-and-there-is... If people are interested, I wrote a pandoc filter to convert such links to things that will continue to work once uploaded to PyPI: https://gist.github.com/cjerdonek/76608610df43fd5b0fc3
in my Makefile: https://github.com/mgedmin/objgraph/blob/master/Makefile#L99
Also, for commands like the latter, is it better to define them in one's setup.py, or simply to have separate scripts in one's repo?
I'm really happy with being able to make releases of any of my packages with
make release
This runs the test suite (tox/detox FTW), checks for uncommitted changes, makes sure MANIFEST.in is correct, makes sure the version number in setup.py matches the version number in CHANGES.rst (and release date is today), makes sure there are no RST errors in long_descrption, and reminds me the commands I need to run to creates a version control system tag and package + upload the sdist to PyPI. (I'm too paranoid to let automated tools perform externally-visible actions like uploading to PyPI or pushing git tags, so I just copy & paste the commands once I'm sure everything's ship-shape.)
That sounds great.
There are other excellent tools for this (e.g. zest.releaser). One day I'll get tired of copy-pasting my Makefiles from project to project and I'll switch to zest.releaser.
Lastly, as these setup-related tasks grow larger and more complicated, I found it helped to break them out into a separate setup package that sits alongside my project's main package library (and even adding tests in some cases). Is this normal? Have other people run into this?
I'm not sure what you mean. Do you have any examples?
I mean that if you are working on project MyProject with package myproject inside the repo, you might find yourself adding ad hoc custom code to setup.py. If this setup.py code starts to grow (a bit like your Makefile may grow), it might make sense to move some of the setup code to a package called something like myproject_setup alongside myproject (which would be used only for setup tasks). And if this setup code was sufficiently complicated, you might find yourself wanting to add unit tests for some of it, so you might have myproject_setup/tests (and even testing it as part of your automated tests, etc). --Chris
Marius Gedminas -- Life begins when you can spend your spare time programming instead of watching television. -- Cal Keegan
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Thu, Oct 02, 2014 at 09:15:54AM -0700, Chris Jerdonek wrote:
On Thu, Oct 2, 2014 at 8:08 AM, Marius Gedminas <marius@pov.lt> wrote:
I wrote restview for a different purpose, but found it rather useful for discovering ReStructuredText problems that would make PyPI's fall back to plaintext rendering.
If you run
restview --long-description
in a directory containing a setup.py, it'll invoke `python setup.py --long-description' and interpret it using the same docutils settings as PyPI, highlighting any errors.
For a more automated (but perhaps less accurate) solution I pipe
python setup.py --long-description | rst2html > /dev/null
Yes, I had been familiar with this latter approach. It is documented here:
https://docs.python.org/2/distutils/packageindex.html#pypi-package-display
It did not work for me in my case though. I eventually found that my use of URL fragments to link to sections elsewhere on the same page, and using relative links (supported by GitHub, https://github.com/blog/1395-relative-links-in-markup-files ) are what broke things.
Incidentally, this gotcha is why restview now has a --pypi-strict mode: https://github.com/mgedmin/restview/issues/18#issuecomment-54122181 (This is enabled by default for restview --long-description.)
Here is a link to a PyPI bug report regarding making it easier to find such issues:
https://bitbucket.org/pypa/pypi/issue/161/rest-formatting-fails-and-there-is...
If people are interested, I wrote a pandoc filter to convert such links to things that will continue to work once uploaded to PyPI:
https://gist.github.com/cjerdonek/76608610df43fd5b0fc3 ...
Lastly, as these setup-related tasks grow larger and more complicated, I found it helped to break them out into a separate setup package that sits alongside my project's main package library (and even adding tests in some cases). Is this normal? Have other people run into this?
I'm not sure what you mean. Do you have any examples?
I mean that if you are working on project MyProject with package myproject inside the repo, you might find yourself adding ad hoc custom code to setup.py. If this setup.py code starts to grow (a bit like your Makefile may grow), it might make sense to move some of the setup code to a package called something like myproject_setup alongside myproject (which would be used only for setup tasks). And if this setup code was sufficiently complicated, you might find yourself wanting to add unit tests for some of it, so you might have myproject_setup/tests (and even testing it as part of your automated tests, etc).
Right. I wanted to see some concrete examples of that code you find putting in your setup.py files. All I've seen before were bits that concatenate README.rst + CHANGES.rst into a long_description, or parse the version number from some source file in the name of DRY. Marius Gedminas -- A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing on usenet?
On Fri, Oct 3, 2014 at 7:23 AM, Marius Gedminas <marius@pov.lt> wrote:
On Thu, Oct 02, 2014 at 09:15:54AM -0700, Chris Jerdonek wrote:
On Thu, Oct 2, 2014 at 8:08 AM, Marius Gedminas <marius@pov.lt> wrote:
Lastly, as these setup-related tasks grow larger and more complicated, I found it helped to break them out into a separate setup package that sits alongside my project's main package library (and even adding tests in some cases). Is this normal? Have other people run into this?
I'm not sure what you mean. Do you have any examples?
I mean that if you are working on project MyProject with package myproject inside the repo, you might find yourself adding ad hoc custom code to setup.py. If this setup.py code starts to grow (a bit like your Makefile may grow), it might make sense to move some of the setup code to a package called something like myproject_setup alongside myproject (which would be used only for setup tasks). And if this setup code was sufficiently complicated, you might find yourself wanting to add unit tests for some of it, so you might have myproject_setup/tests (and even testing it as part of your automated tests, etc).
Right. I wanted to see some concrete examples of that code you find putting in your setup.py files.
All I've seen before were bits that concatenate README.rst + CHANGES.rst into a long_description, or parse the version number from some source file in the name of DRY.
In some older projects, I had a fair amount of code that essentially did what "check-manifest" does (but as part of running sdist). And yes, there was also code to do some pre-processing of README.md and to convert that with other files into a .rst long_description. In a newer project, I'm starting to add additional code to support running pandoc, which you can see here, for example: https://github.com/cjerdonek/open-rcv/blob/master/openrcv_setup/pandoc.py I use pandoc to convert files like README.md into both rst files for the long_description, and into HTML for local viewing (a bit similar to what restview does). I'm not sure what other tools people use for this, but I've been happy with pandoc, and it is quite versatile and powerful. I can point you to additional specific examples off-list if you want. --Chris
participants (4)
-
Carl Meyer -
Chris Jerdonek -
Donald Stufft -
Marius Gedminas