setup.py + cython == chicken and the egg problem

[Dan Stromberg]

On Wed, Aug 17, 2022 at 10:20 AM Christian Gollwitzer <auriocus at gmx.de>
wrote:

> Am 16.08.22 um 23:03 schrieb Dan Stromberg:
> > I'm attempting to package up a python package that uses Cython.
> >
> > Rather than build binaries for everything under the sun, I've been
> focusing
> > on including the .pyx file and running cython on it at install time.
> This
> > requires a C compiler, but I'm OK with that.
> >
> > However, when I try to install the package from test.pypi.org, I get:
> > $ python3 -m pip install -i https://test.pypi.org/simple/ pyx-treap
> > below cmd output started 2022 Tue Aug 16 01:55:16 PM PDT
> > Looking in indexes: https://test.pypi.org/simple/
> > Collecting pyx-treap
> >    Downloading
> >
> https://test-files.pythonhosted.org/packages/3a/41/af5360934adccfc086a39e1f720323895144b53454ff6dacc0f06267db55/pyx_treap-2.0.15.tar.gz
> > (125 kB)
> >
> >
>  ????????????????????????????????????????????????????????????????????????????????
> > 125.9/125.9 kB 1.9 MB/s eta 0:00:00
> >    Installing build dependencies ... error
> >    error: subprocess-exited-with-error
> >
> >    ×? pip subprocess to install build dependencies did not run
> successfully.
> >    ?? exit code: 1
> >    ????> [3 lines of output]
> >        Looking in indexes: https://test.pypi.org/simple/
> >        ERROR: Could not find a version that satisfies the requirement
> > setuptools (from versions: none)
> >        ERROR: No matching distribution found for setuptools
> >        [end of output]
> >
> >    note: This error originates from a subprocess, and is likely not a
> > problem with pip.
> > error: subprocess-exited-with-error
>
>
> I looked at your code and I think you are trying too hard. As far as I
> understand, you need Cython to be installed before the build process
> begins. Your entry in pyproject.toml should take care of that.
> But you also have these lines in your setup.py
>
> subprocess.check_call('%s -m pip install cython' % (sys.executable, ),
> shell=True)
> subprocess.check_call('%s -m cython pyx_treap.pyx' % (sys.executable, ),
> shell=True)
>
> The first one calls out to pip while pip is already running, I'm not
> sure that this will work, but judging from the error message it is
> looking for the requirements also from test.pypi. Maybe this is the
> reason that it fails (the error message says that it can't find
> setuptools). So jut delete this line and it might already work
>
> The second line, which compiles the Cython code, also runs *at every
> invocation of setup.py*, even if you'd do just
>
>         python3 setup.py --help
>
> It may still work, but the correct way to do it is to create a build
> extension for setuptools. In my project you can see this here:
>
>         https://github.com/j-from-b/CDEF/blob/main/setup.py#L88
>
> OTOH, I would be surprised if Cython did not have this already, indeed
> you imported cythonize from Cython.Build. So maybe just deleting these
> two lines and it might work?
>

I commented out those too lines, but I'm still getting errors.  They seem
to stem from:
$ "/home/dstromberg/venv/pyx-treap-testing/bin/python3",
["/home/dstromberg/venv/pyx-treap-testing/bin/python3",
"/home/dstromberg/venv/pyx-treap-testing/lib/python3.9/site-packages/pip/__pip-runner__.py",
"install", "--ignore-installed", "--no-user", "--prefix",
"/tmp/pip-build-env-9_ivrsb6/overlay", "--no-warn-script-location",
"--no-binary", ":none:", "--only-binary", ":none:", "-i", "
https://test.pypi.org/simple/", "--", "setuptools >= 44.1.1", "wheel",
"Cython"]
"/home/dstromberg/venv/pyx-treap-testing/bin/python3"
"/home/dstromberg/venv/pyx-treap-testing/lib/python3.9/site-packages/pip/__pip-runner__.py"
"install" "--ignore-installed" "--no-user" "--prefix"
"/tmp/pip-build-env-9_ivrsb6/overlay" "--no-warn-script-location"
"--no-binary" ":none:" "--only-binary" ":none:" "-i" "
https://test.pypi.org/simple/" "--" "setuptools >= 44.1.1" "wheel" "Cython"
Looking in indexes: https://test.pypi.org/simple/
ERROR: Could not find a version that satisfies the requirement
setuptools>=44.1.1 (from versions: none)
ERROR: No matching distribution found for setuptools>=44.1.1

I copied that out of an strace.

That's likely related to my pyproject.toml:
$ cat pyproject.toml
below cmd output started 2022 Wed Aug 17 01:57:09 PM PDT
[build-system]
requires = ["setuptools >= 44.1.1", "wheel", "Cython"]
build-backend = "setuptools.build_meta"

Any other suggestions folks?