Using Makefiles in Python projects

[Cameron Simpson]

On 07Nov2019 22:20, Vitaly Potyarkin <sio.wtf at gmail.com> wrote:
>What do you think of using Makefiles for automating common chores in
>Python projects? Like linting, type checking and testing?

I do use one for some things. (Not linting, which I'll describe lower 
down.)

I do like to use it for what make's strength is: running actions based 
on updated files, but there are a few "action" targets too.

Here's a slightly edited and trimmed excerpt from a Makefile (well, 
Mykefile, but the intent and syntax are very similar) from a current 
project:

    venv_dir = $./venv
    venv_reqs = requirements.txt
    venv_pip = $(venv_dir)/bin/pip

    py_static_bundle = $(py_app)/static/app.js

    fleet_dev = [ "$$HOST" = fleet -a "$$USER" = cameron ]

    _help:
            @echo '_build: make $(py_static_bundle)'
            @echo '_deploy_tip: formally deploy the current tip to the dev host dev tree:'
            @echo '_sync_dev: rsync the current working files into the dev tip tree'

    _build:
            make $(py_static_bundle)

    _venv:
            :make $(venv_dir)

    $(venv_dir):    $(venv_reqs)
            python3 -m venv $(venv_dir)
            $(venv_pip) install -U pip
            $(venv_pip) install -U -r $(venv_reqs)

    $(py_static_bundle): $(app_js)
            env-dev browserify -t [ babelify --presets [ react ] ] $? >$@

    _sync_dev:
            $(fleet_dev) || { echo "not cameron at fleet, refusing to rsync" >&2; exit 1; }
            :make $(py_static_bundle)
            $(sync_src(target_dev_host:releases/tip))

Things to note:

"Virtual targets" are actions rather than result files, and start with 
an underscore.

The default target is _help, which recites a description of the other 
targets.

The _venv target makes the venv directory if missing, and that runs 
python3 -m venv and then pip to do the required packages.

There's a traditional make rule i.e. an action based on a dependent 
file, gosh! for the static bundle (javascript compiled into a single 
file).

You'll see that _sync_dev has a leading sanity check: it won't run 
unless I'm me on my personal machine; avoids accidents. This is 
important because it isn't running the formal deploy script, which 
builds a clean versioned tree at the far end, instead it just rsyncs my 
local code into a "live" dev tree at the far end. Obviously that is a 
very personal hack.

>I've come up with a reusable Makefile for automating virtual 
>environment
>management in Python projects. I think it can be useful for simplifying
>the onboarding of new developers (both new to project and new to Python)
>and for documenting project's development practices.

These are laudable goals. I like the expose the dev practices in 
particular.

>Here it is:
>- Repo: https://github.com/sio/Makefile.venv
>- Demo screencast: https://asciinema.org/a/279646
>
>What do you think? Is this useful or I'm just unaware of some tool that
>abstracts venv chores away better?

There may be other tools.

Linting can be personal. In my current team we have varying lint habits 
among the devs. OTOH a common "required lint before release" could be 
useful, covering off the project lint standards if they exist.

I lint by hand, with 2 main levels of automation:

1: I have a personal "lint" script with runs my preferred linters with 
my preferred options, so I can just say "lint filenames..." and have it 
do the right thing. It is also multilanguage: python linters, shell 
linters, etc.

2: I have a few VCS related shell functions. One is named "lint". With 
arguments is invokes the above lint script directly. _Without_ arguments 
it lints the "modified" (== changed and not committed) files. So I can 
just say "lint" and it lints my changes, not all the files.

Cheers,
Cameron Simpson <cs at cskk.id.au>