I've been working on ensconced which lets you use a SConstruct to build. It is easier than customizing distutils. https://github.com/dholth/pysdl2-cffi/blob/master/SConstruct is the most code-generation-y project. Its build might be a little out of date and it has an independent setup.py building the project with setuptools. https://github.com/dholth/bioopenssl doesn't have C extensions but is an example of a stub setup.py (only necessary for the develop command) and pyproject.toml (which pip can use without a setup.py via PEP 517). On Fri, Dec 11, 2020 at 10:09 AM Julian Smith wrote:

Hello

I have a project with a fairly involved build process where we generate C++ via a python-clang-based code analyser, generates Python bindings for the resulting C++ API with SWIG, and finally compile and link to create various .so's and .py files.

I'd like to package things up with a setup.py script, but from what i have read, it looks like distutils expects to compile, link and run swig itself, using distutils.core.setup()'s ext_modules arg?

Is there a way to write setup.py so that it instead runs an external command (e.g. 'cd foo && make all') when it is asked to build?

Thanks for any help,

- Jules

-- http://op59.net

-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/AJDJU...

enscons is an alternative to distutils.

You would have to have a high tolerance for learning SCons. I'm aware that this is not for everyone. Then you could write a SConstruct with dependent tasks in a normal build system way. e.g. target = env.Command("a task", ...) platlib = env.Whl("platlib", target, root=".") whl = env.WhlFile(platlib) and have each step re-build only when its sources change. Here is a stackoverflow answer about custom distutils commands. https://stackoverflow.com/questions/1710839/custom-distutils-commands

I've recently returned to this issue after a while away. I've ended up with a custom setup.py that runs the external build system as required, and allows pip install/sdist/bdist_wheel to work. However as a result of all this. i have some general questions. Basically i'm confused about the lack of information that is available on how to write a setup.py for packaging a project. If a project is pure python, or it uses compiled code and it's possible to specify how to build it using the swig/compile/link facilities provided by distutils or setuptools, then it's easy enough to use distutils or setuptools. However if, like me, one needs to use an external build system, it's very difficult to figure out what to do. There seem to be two approaches: Approach 1 is to pass callbacks to distutils.core.setup() or setuptools.setup(). However there doesn't appear to be documentation in either of these modules about what such callbacks should do or how/when they are called. The only way to figure things out seems to be to look at the distutils or setuptools source, which i'm finding pretty opaque. As far as i can tell, callbacks are only given information about the original command line arguments rather than more abstract information such as where to put output files, so distutils and setuptools don't seem to be giving any added value here. Approach 2 is to give up on distutils and setuptools, and instead write setup.py from scratch, directly handling the command-line arguments from pip. Given how fundamental the pip tool is to Python packaging, i was hoping that the command-line arguments that pip passes to setup.py would be standardised and documented, but this doesn't seem to be the case. One can gather some information by running pip commands and have setup.py print out sys.argv, but this is hardly a robust way of doing things. For example there's no guarantee that future versions of pip won't start using different command-line arguments that setuptools already knows about. So as far as i can tell, there are two levels of abstraction at which on can implement customised Python packaging (the setuptools.setup()'s callbacks or the setup.py command line), but neither one seems to be documented or standardised. Is that right? Or am i missing something fundamental here? Many thanks for any help. - Jules On Mon, 14 Dec 2020 13:29:28 -0500 Daniel Holth wrote:

You would have to have a high tolerance for learning SCons. I'm aware that this is not for everyone. Then you could write a SConstruct with dependent tasks in a normal build system way. e.g.

target = env.Command("a task", ...) platlib = env.Whl("platlib", target, root=".") whl = env.WhlFile(platlib)

and have each step re-build only when its sources change.

Here is a stackoverflow answer about custom distutils commands. https://stackoverflow.com/questions/1710839/custom-distutils-commands

-- http://op59.net

On 23 Mar 2021, at 19:13, Julian Smith wrote:

Approach 1 is to pass callbacks to distutils.core.setup() or setuptools.setup(). However there doesn't appear to be documentation in either of these modules about what such callbacks should do or how/when they are called. The only way to figure things out seems to be to look at the distutils or setuptools source, which i'm finding pretty opaque.

[snipped]

As far as i can tell, callbacks are only given information about the original command line arguments rather than more abstract information such as where to put output files, so distutils and setuptools don't seem to be giving any added value here.

[snipped]

Given how fundamental the pip tool is to Python packaging, i was hoping that the command-line arguments that pip passes to setup.py would be standardised and documented, but this doesn't seem to be the case.

What you get from pip is standardised and documented in PEP 517. The part that’s not so is what happens *after* setup tools receives those things, which is entirely setuptools internals. [snipped]

So as far as i can tell, there are two levels of abstraction at which on can implement customised Python packaging (the setuptools.setup()'s callbacks or the setup.py command line), but neither one seems to be documented or standardised.

Is that right? Or am i missing something fundamental here?

You’re basically correct, there are two abstractions in play. The first (between pip and setuptools) is documented. The second (between setuptools and the external command) is not, because that’s setuptools internals and setuptools doesn’t really expect people to mess with them. You still can if you want to, of course, but you’re on your own. So my advice would be to ditch setup tools entirely and work directly with the PEP 517 interface. You don’t need to start from scratch though; there are several PEP 517 implementations other than setuptools, and some of them already implemented some mechanisms to call external build commands, such as enscons[1] (via SCons) and PDM[2] (via setuptools). Some other projects also have already expressed interests supporting this, such as Flit[3], and I think they’d be more than happy to discuss this if you want to work this with them. [1]: https://github.com/dholth/enscons [2]: https://github.com/frostming/pdm [3]: https://github.com/takluyver/flit -- Tzu-ping Chung (@uranusjr) uranusjr@gmail.com https://uranusjr.com

On Tue, 23 Mar 2021 12:34:47 +0000 "Thomas Kluyver" wrote:

On Tue, 23 Mar 2021, at 11:13, Julian Smith wrote:

...

So as far as i can tell, there are two levels of abstraction at which on can implement customised Python packaging (the setuptools.setup()'s callbacks or the setup.py command line), but neither one seems to be documented or standardised.

That's right. PEP 517 (https://www.python.org/dev/peps/pep-0517/ ) is our effort to have a standardised, documented interface to replace running setup.py. setup.py is now run by setuptools as part of a PEP 517 backend.

Many thanks for this, and the other replies from Bernat Gabor and Tzu-ping Chung . I had already come across PEP-517 actually, but hadn't appreciated its importance. I think i've managed to get my packaging to work using PEP-517 callbacks. I was a little surprised to find out that one can't use pip to create an sdist, but i'm a bit late to this party and it looks like there's been plenty of discussion about this in the past. I've ended up with a pyproject.toml that points to a setup.py file which functions both as a PEP-517 module (providing functions build_wheel() and (unused) build_sdist()), and also as a conventional setup.py script for things like './setup.py sdist'. I've come across a problem where an old pip-18.1 fails to install from an sdist or wheel, with error: ERROR: You must give at least one requirement to install (see "pip help install") pip-20.2.1 and pip-21.0.1 work fine, including installing from an sdist on test.py.pi. Should i be worried about this? I'd like my package to install on anyone's machine, regardless of their pip version. But maybe users are expected to always run a recent pip version? After all it's easy enough to upgrade it. Thanks, - Jules -- http://op59.net

On Thu, 25 Mar 2021, at 19:07, Julian Smith wrote:

I was a little surprised to find out that one can't use pip to create an sdist, but i'm a bit late to this party and it looks like there's been plenty of discussion about this in the past.

There is now the (quite new) 'build' tool which can be used to make both sdists and wheels by working with a PEP 517 backend: https://pypi.org/project/build/

I've come across a problem where an old pip-18.1 fails to install from an sdist or wheel, with error:

I'm surprised it fails on the wheels - how exactly are you using pip? If you're relying on PEP 517, then pip 18.1 is too old to install it from an sdist - PEP 517 support was added in 19.0 (https://pip.pypa.io/en/stable/news/#id448 ). But that shouldn't be a problem for installing from a wheel that has already been made. 18.1 is over 2 years outdated, so I think it's reasonable to ask people to upgrade, but you may well still encounter people who have older versions, mostly where they've got it through a Linux distro package manager. Thomas

On Thu, 25 Mar 2021 20:01:12 +0000 "Thomas Kluyver" wrote:

On Thu, 25 Mar 2021, at 19:07, Julian Smith wrote:

...

I was a little surprised to find out that one can't use pip to create an sdist, but i'm a bit late to this party and it looks like there's been plenty of discussion about this in the past.

There is now the (quite new) 'build' tool which can be used to make both sdists and wheels by working with a PEP 517 backend: https://pypi.org/project/build/

Thanks for this, i tried it - see below.

...

I've come across a problem where an old pip-18.1 fails to install from an sdist or wheel, with error:

I'm surprised it fails on the wheels - how exactly are you using pip?

Ah, sorry, i was confused, installing from a wheel works. It's just installing from a local or remote sdist that fails. For example with pip-18.1: pip -vvv install -i https://test.pypi.org/simple optest We get: Created temporary directory: /tmp/pip-ephem-wheel-cache-n_a2wzyf Created temporary directory: /tmp/pip-req-tracker-6oichqha Created requirements tracker '/tmp/pip-req-tracker-6oichqha' Created temporary directory: /tmp/pip-install-kep8x9wu Looking in indexes: https://test.pypi.org/simple Collecting optest 1 location(s) to search for versions of optest: * https://test.pypi.org/simple/optest/ Getting page https://test.pypi.org/simple/optest/ Looking up "https://test.pypi.org/simple/optest/" in the cache Current age based on date: 8 Freshness lifetime from max-age: 600 Freshness lifetime from request max-age: 0 Starting new HTTPS connection (1): test.pypi.org:443 https://test.pypi.org:443 "GET /simple/optest/ HTTP/1.1" 304 0 Analyzing links from page https://test.pypi.org/simple/optest/ Found link https://test-files.pythonhosted.org/packages/20/53/b75897caba2b7249802439ab0... (from https://test.pypi.org/simple/optest/), version: 1.0.20210326.2051 Using version 1.0.20210326.2051 (newest of versions: 1.0.20210326.2051) Created temporary directory: /tmp/pip-unpack-v76_dlvl Looking up "https://test-files.pythonhosted.org/packages/20/53/b75897caba2b7249802439ab0..." in the cache Current age based on date: 7 Freshness lifetime from max-age: 0 Starting new HTTPS connection (1): test-files.pythonhosted.org:443 https://test-files.pythonhosted.org:443 "GET /packages/20/53/b75897caba2b7249802439ab0fc22562ae63e75de095054400dda7a94dc8/optest-1.0.20210326.2051.tar.gz HTTP/1.1" 200 6951 Downloading https://test-files.pythonhosted.org/packages/20/53/b75897caba2b7249802439ab0... Downloading from URL https://test-files.pythonhosted.org/packages/20/53/b75897caba2b7249802439ab0... (from https://test.pypi.org/simple/optest/) Updating cache with response from "https://test-files.pythonhosted.org/packages/20/53/b75897caba2b7249802439ab0..." Caching due to etag Added optest from https://test-files.pythonhosted.org/packages/20/53/b75897caba2b7249802439ab0... to build tracker '/tmp/pip-req-tracker-6oichqha' Created temporary directory: /tmp/pip-build-env-yx5wdf16 Installing build dependencies: started Running command /home/jules/pylocal/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-yx5wdf16 --no-warn-script-location -v --no-binary :none: --only-binary :none: -i https://test.pypi.org/simple -- Created temporary directory: /tmp/pip-ephem-wheel-cache-3hs3xx6m Re-using requirements tracker '/tmp/pip-req-tracker-6oichqha' Created temporary directory: /tmp/pip-install-xif4sjdq Cleaning up... Cleaned build tracker '/tmp/pip-req-tracker-6oichqha' ERROR: You must give at least one requirement to install (see "pip help install") Exception information: Traceback (most recent call last): File "/home/jules/pylocal/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 143, in main status = self.run(options, args) File "/home/jules/pylocal/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 313, in run self.name, wheel_cache File "/home/jules/pylocal/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 252, in populate_requirement_set '(see "pip help %(name)s")' % opts) pip._internal.exceptions.CommandError: You must give at least one requirement to install (see "pip help install") Installing build dependencies: finished with status 'error' Cleaning up... Removing source in /tmp/pip-install-kep8x9wu/optest Removed optest from https://test-files.pythonhosted.org/packages/20/53/b75897caba2b7249802439ab0... from build tracker '/tmp/pip-req-tracker-6oichqha' Removed build tracker '/tmp/pip-req-tracker-6oichqha' Command "/home/jules/pylocal/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-yx5wdf16 --no-warn-script-location -v --no-binary :none: --only-binary :none: -i https://test.pypi.org/simple --" failed with error code 1 in None Exception information: Traceback (most recent call last): File "/home/jules/pylocal/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 143, in main status = self.run(options, args) File "/home/jules/pylocal/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 338, in run resolver.resolve(requirement_set) File "/home/jules/pylocal/lib/python3.7/site-packages/pip/_internal/resolve.py", line 102, in resolve self._resolve_one(requirement_set, req) File "/home/jules/pylocal/lib/python3.7/site-packages/pip/_internal/resolve.py", line 256, in _resolve_one abstract_dist = self._get_abstract_dist_for(req_to_install) File "/home/jules/pylocal/lib/python3.7/site-packages/pip/_internal/resolve.py", line 209, in _get_abstract_dist_for self.require_hashes File "/home/jules/pylocal/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 298, in prepare_linked_requirement abstract_dist.prep_for_dist(finder, self.build_isolation) File "/home/jules/pylocal/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 109, in prep_for_dist "Installing build dependencies" File "/home/jules/pylocal/lib/python3.7/site-packages/pip/_internal/build_env.py", line 122, in install_requirements call_subprocess(args, show_stdout=False, spinner=spinner) File "/home/jules/pylocal/lib/python3.7/site-packages/pip/_internal/utils/misc.py", line 723, in call_subprocess % (command_desc, proc.returncode, cwd)) pip._internal.exceptions.InstallationError: Command "/home/jules/pylocal/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-yx5wdf16 --no-warn-script-location -v --no-binary :none: --only-binary :none: -i https://test.pypi.org/simple --" failed with error code 1 in None After 'pip install --upgrade pip' upgrades to pip-21.0.1, it installs fine: pip install -i https://test.pypi.org/simple optest Looking in indexes: https://test.pypi.org/simple Collecting optest Using cached optest-1.0.20210326.2051-py3-none-linux_x86_64.whl Installing collected packages: optest Successfully installed optest-1.0.20210326.2051 I can't tell from pip-18.1's diagnostics what exactly is going wrong. Given that my setup.py implements the normal distutils-style argv handling, i was expecting things to work, but pip-18.1 appears to fail before it even tries to run setup.py, so the only thing i can think of is that the PKG-INFO file could be causing a problem? I've also tried building an sdist with the https://pypi.org/project/build/ project's --sdist option; this sdist fails to install with pip-18.1 in the same way. Thanks, - Julian -- http://op59.net

On Fri, 26 Mar 2021 23:19:30 +0000 "Thomas Kluyver" wrote:

On Fri, 26 Mar 2021, at 23:04, Julian Smith wrote:

...

I can't tell from pip-18.1's diagnostics what exactly is going wrong.

Given that my setup.py implements the normal distutils-style argv handling, i was expecting things to work, but pip-18.1 appears to fail before it even tries to run setup.py, so the only thing i can think of is that the PKG-INFO file could be causing a problem?

Looking at your package, I think the issue is that you have the files (setup.py etc.) in the top level of the tarball, but pip is expecting a <name>-<version> folder in the tarball which contains all the files.

The actual format of sdists is something that's not yet properly documented, as far as I know, but you can look at essentially any package to see what I mean in this case. E.g. six:

https://files.pythonhosted.org/packages/6b/34/415834bfdafca3c5f451532e8a8d9b...

Thanks for this. I had high hopes that this would solve things, but unfortunately i'm seeing the same behaviour as before. I've uploaded my new sdist, optest-1.0.20210326.2328.tar.gz, so one can attempt to install it with: pip install -i https://test.pypi.org/simple optest The six-1.15.0.tar.gz file you mention does install with pip-18.1, in an interesting way - it fails at first before eventually succeeding by running 'setup.py install': Processing ./six-1.15.0.tar.gz Building wheels for collected packages: six Running setup.py bdist_wheel for six ... error Complete output from command /home/jules/artifex-remote/pylocal/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-req-build-naagtvzk/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-ouokji77 --python-tag cp37: usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: -c --help [cmd1 cmd2 ...] or: -c --help-commands or: -c cmd --help error: invalid command 'bdist_wheel' ---------------------------------------- Failed building wheel for six Running setup.py clean for six Failed to build six Installing collected packages: six Running setup.py install for six ... done Successfully installed six-1.15.0 This is how i thought my sdist would install, so i guess i could try to move my sdist to be similar to six-1.15.0.tar.gz, and things should eventually start to work. Thanks, - Jules -- http://op59.net

On Sun, 28 Mar 2021 18:13:26 +0100 Julian Smith wrote:

On Sat, 27 Mar 2021 00:05:35 +0000 Julian Smith wrote:

I eventually managed to local installs from sdist to work with pip-18.1 by modifying my pyproject.toml, changing:

[build-system] requires = []

to: [build-system] requires = ["wheel"]

[Using "setup" also works.]

This seems to make a difference even though my setup.py and pipcl.py files don't use setuptools.

I also had to make my setup.py script handle the 'dist_info' command. setuptools.setup()'s help text doesn't mention this so it's not immediately clear what it is supposed to do, but it turns out to be similar to egg_info except for putting files in /<name>.dist-info/ instead of /.egg-info/.

Unfortunately although i can now install from a local sdist with old and new pip, the 'requires = ["wheel"]' seems to break things when testing on test.pypi.org - i get:

requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://test.pypi.org/simple/wheel/

pip-18.1 then gives up. pip-21.0.1 carries on, running setup.py directly to do the install of my package.

I guess i could try uploading to pypi.org, and see whether it can give the dummy "wheel" package, but i was hoping to be able to get things working reliably on test.pypi.org first.

In the end i decided not to try to support installation with old pip-18.1. So i've now uploaded the project to pypi.org. It's a set of Python bindings for the PDF library MuPDF, and can be installed with: pip install mupdf The install takes a few minutes to compile original source and run SWIG etc. More information is at: https://twiki.ghostscript.com/do/view/Main/MuPDFWrap If anyone is interested, here's the MuPDF commit that adds support for Python packaging, adding a Python module pipcl.py that helps to provide generic PEP-517 support and basic setup.py command-line usage: https://git.ghostscript.com/?p=mupdf.git;a=commitdiff;h=153c67ddf5 Thanks, - Julian -- http://op59.net