setup.py + cython == chicken and the egg problem

[Dan Stromberg]

On Wed, Aug 17, 2022 at 1:58 PM Dan Stromberg <drsalists at gmail.com> wrote:

> On Wed, Aug 17, 2022 at 10:20 AM Christian Gollwitzer <auriocus at gmx.de>
> wrote:
>
>> Am 16.08.22 um 23:03 schrieb Dan Stromberg:
>> > I'm attempting to package up a python package that uses Cython.
>> >
>> > Rather than build binaries for everything under the sun, I've been
>> focusing
>> > on including the .pyx file and running cython on it at install time.
>> This
>> > requires a C compiler, but I'm OK with that.
>> >
>> > However, when I try to install the package from test.pypi.org, I get:
>> > $ python3 -m pip install -i https://test.pypi.org/simple/ pyx-treap
>> > below cmd output started 2022 Tue Aug 16 01:55:16 PM PDT
>> > Looking in indexes: https://test.pypi.org/simple/
>> > Collecting pyx-treap
>> >    Downloading
>> >
>> https://test-files.pythonhosted.org/packages/3a/41/af5360934adccfc086a39e1f720323895144b53454ff6dacc0f06267db55/pyx_treap-2.0.15.tar.gz
>> > (125 kB)
>> >
>> >
>>  ????????????????????????????????????????????????????????????????????????????????
>> > 125.9/125.9 kB 1.9 MB/s eta 0:00:00
>> >    Installing build dependencies ... error
>> >    error: subprocess-exited-with-error
>> >
>> >    ×? pip subprocess to install build dependencies did not run
>> successfully.
>> >    ?? exit code: 1
>> >    ????> [3 lines of output]
>> >        Looking in indexes: https://test.pypi.org/simple/
>> >        ERROR: Could not find a version that satisfies the requirement
>> > setuptools (from versions: none)
>> >        ERROR: No matching distribution found for setuptools
>> >        [end of output]
>> >
>> >    note: This error originates from a subprocess, and is likely not a
>> > problem with pip.
>> > error: subprocess-exited-with-error
>>
>>
>> I looked at your code and I think you are trying too hard. As far as I
>> understand, you need Cython to be installed before the build process
>> begins. Your entry in pyproject.toml should take care of that.
>> But you also have these lines in your setup.py
>>
>> subprocess.check_call('%s -m pip install cython' % (sys.executable, ),
>> shell=True)
>> subprocess.check_call('%s -m cython pyx_treap.pyx' % (sys.executable, ),
>> shell=True)
>>
>> The first one calls out to pip while pip is already running, I'm not
>> sure that this will work, but judging from the error message it is
>> looking for the requirements also from test.pypi. Maybe this is the
>> reason that it fails (the error message says that it can't find
>> setuptools). So jut delete this line and it might already work
>>
>> The second line, which compiles the Cython code, also runs *at every
>> invocation of setup.py*, even if you'd do just
>>
>>         python3 setup.py --help
>>
>> It may still work, but the correct way to do it is to create a build
>> extension for setuptools. In my project you can see this here:
>>
>>         https://github.com/j-from-b/CDEF/blob/main/setup.py#L88
>>
>> OTOH, I would be surprised if Cython did not have this already, indeed
>> you imported cythonize from Cython.Build. So maybe just deleting these
>> two lines and it might work?
>>
>
> I commented out those too lines, but I'm still getting errors.  They seem
> to stem from:
> $ "/home/dstromberg/venv/pyx-treap-testing/bin/python3",
> ["/home/dstromberg/venv/pyx-treap-testing/bin/python3",
> "/home/dstromberg/venv/pyx-treap-testing/lib/python3.9/site-packages/pip/__pip-runner__.py",
> "install", "--ignore-installed", "--no-user", "--prefix",
> "/tmp/pip-build-env-9_ivrsb6/overlay", "--no-warn-script-location",
> "--no-binary", ":none:", "--only-binary", ":none:", "-i", "
> https://test.pypi.org/simple/", "--", "setuptools >= 44.1.1", "wheel",
> "Cython"]
> "/home/dstromberg/venv/pyx-treap-testing/bin/python3"
> "/home/dstromberg/venv/pyx-treap-testing/lib/python3.9/site-packages/pip/__pip-runner__.py"
> "install" "--ignore-installed" "--no-user" "--prefix"
> "/tmp/pip-build-env-9_ivrsb6/overlay" "--no-warn-script-location"
> "--no-binary" ":none:" "--only-binary" ":none:" "-i" "
> https://test.pypi.org/simple/" "--" "setuptools >= 44.1.1" "wheel"
> "Cython"
> Looking in indexes: https://test.pypi.org/simple/
> ERROR: Could not find a version that satisfies the requirement
> setuptools>=44.1.1 (from versions: none)
> ERROR: No matching distribution found for setuptools>=44.1.1
>
> I copied that out of an strace.
>
> That's likely related to my pyproject.toml:
> $ cat pyproject.toml
> below cmd output started 2022 Wed Aug 17 01:57:09 PM PDT
> [build-system]
> requires = ["setuptools >= 44.1.1", "wheel", "Cython"]
> build-backend = "setuptools.build_meta"
>
> Any other suggestions folks?
>

I don't know why, but if I delete the --ignore-installed option, I don't
get the error:
$ "/home/dstromberg/venv/pyx-treap-testing/bin/python3"
"/home/dstromberg/venv/pyx-treap-testing/lib/python3.9/site-packages/pip/__pip-runner__.py"
"install" "--ignore-installed" "--prefix"
"/tmp/pip-build-env-9_ivrsb6/overlay" "-i" "https://test.pypi.org/simple/"
"--" "setuptools>=44.1.1"
below cmd output started 2022 Wed Aug 17 02:56:24 PM PDT
Looking in indexes: https://test.pypi.org/simple/
ERROR: Could not find a version that satisfies the requirement
setuptools>=44.1.1 (from versions: none)
ERROR: No matching distribution found for setuptools>=44.1.1
(setuptools-investigation) above cmd output done    2022 Wed Aug 17
02:56:24 PM PDT
dstromberg at tp-mini-c:~ x86_64-pc-linux-gnu 2995

$ "/home/dstromberg/venv/pyx-treap-testing/bin/python3"
"/home/dstromberg/venv/pyx-treap-testing/lib/python3.9/site-packages/pip/__pip-runner__.py"
"install" "--prefix" "/tmp/pip-build-env-9_ivrsb6/overlay" "-i" "
https://test.pypi.org/simple/" "--" "setuptools>=44.1.1"
below cmd output started 2022 Wed Aug 17 02:56:35 PM PDT
Looking in indexes: https://test.pypi.org/simple/
Requirement already satisfied: setuptools>=44.1.1 in
./venv/pyx-treap-testing/lib/python3.9/site-packages (63.4.1)

If I search for foo on pypi and testpypi, shouldn't I get foo before
foo-bar and bar-foo?

Because if I search for setuptools on pypi, I get setuptools as my first
hit, but on testpypi, I don't see setuptools anywhere in the first page.
That's perhaps significant if the search ordering is working as described
above.

?