By splitting it into two different packages you end up with the same situation that currently plagues psycopg2/psycopg2-binary whereby if you depend on psycopg2 you can't easily swap in psycopg2-binary and vice-versa as the two don't satisfy the same dependency. Number 3 is kind of what sqlalchemy does, and then provide wheels for a huge variety of platforms to allow people to install the package without needing a compiler themselves. Bert

On Aug 2, 2020, at 04:42, Daniele Varrazzo wrote:

Hello,

packaging psycopg3, I'm wondering what is the best way to provide an optional optimisation module.

1) provide a separate psycopg3-c distribution 2) provide an extra psycopg3[c] 3) try building the extension and fail quietly.

1) seems the cleanest approach: the psycopg3 distribution would have no build-time external dependency (Cython, -dev packages, a compiler) and psycopg3-c can fail hard if some of these dependencies are missing. I am currently trying this approach, finding some problems in working out a good files layout to have two setup.py in the same git repository.

2) would be nice but I don't see a way to identify the extra requested at build time to implement a build_ext command such that if "c" is not the extra then don't do anything. it seems that extra are thought for a different use case, not for optional build-time parts

3) would give me endless headaches to work out why something failed, differentiate failures for missing dependencies from real errors, and dealing with user reports.

What would be your advice? Press on with 1 or a different approach? Examples are welcome (the only one I have in mind is PyYAML doing 3).

-- Daniele -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/GEPU2...

Ideally this problem should be solved with the Provides-Dist metadata.[1] The exact semantic to the flag is unfortunately not discussed much currently, and no installer implementations that I know of (including pip) support it at all. This also recently came up in pip’s issue tracker,[2] and I guess this is as good a time as any to start the conversation if someone is willing to help drive that discussion forward. [1]: https://packaging.python.org/specifications/core-metadata/#provides-dist-mul... [2]: https://github.com/pypa/pip/issues/8669 -- Tzu-ping Chung (@uranusjr) uranusjr@gmail.com https://uranusjr.com

On 03/8, 2020, at 05:59, Daniele Varrazzo wrote:

On Sun, 2 Aug 2020 at 21:34, Bert JW Regeer wrote:

...

By splitting it into two different packages you end up with the same situation that currently plagues psycopg2/psycopg2-binary whereby if you depend on psycopg2 you can't easily swap in psycopg2-binary and vice-versa as the two don't satisfy the same dependency.

The psycopg2 wheel or not wheel is the situation you describe, and it's really suboptimal from the point of declaring dependencies: beginners would like to use psycopg2-binary, but projects are advised to depend on psycopg2, so they either don't get the wheel benefit or they end up in a tangle of dependencies, two distributions installing the same files, bad stuff. Offering the C distribution as an opt-in extension would allow projects to depend only on the pure python psycopg3, which would be also the right choice for beginners, and allowing the grown-ups with a compiler to go faster by installing psycopg3-c too, which wouldn't conflict with the basic package.

...

Number 3 is kind of what sqlalchemy does, and then provide wheels for a huge variety of platforms to allow people to install the package without needing a compiler themselves.

The difference in performance of the C extension is important enough (15-20x - https://www.varrazzo.com/blog/2020/05/19/a-trip-into-optimisation/) to arguably make or break a deal. If someone wanted the C extension because they need the performance I wouldn't want its installation to fail silently.

-- Daniele -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/MQNGT...