Re: Capture manylinux environment set-up in pyproject.toml
Its interesting for me, but I see some problems. System_dependencies: many linux have different names (example: apache2 vs httpd). Sometimes same dependency is sliced in multiple packages (example: python3.9 depends on python3.9-minimal and...). How many times have you see a machine with python but without idle or tkinter? And naming convention is a pain for build systems dependencies and runtime system dependencies. Fortunately we have containers and can run different linux flavours. And old version of theese flavour which is very important. Environment variables: Usually used to inject configuration. My GOPATH env var is under my home. Maybe a env name can have a description, default value,... but not a fixed value, because is not easy to share with others. To avoid system dependencies, some build tools like https://github.com/joerick/cibuildwheel are centered on CI suites. With a CI you can use windows, linux, osx,... without think in mount virtual machines (libvirt, virtualbox,...) or coordinate buildbots farms more or less well configured. Once a wheel is compiled, build dependencies are no needed more. Then, cibuildwheel uses other libraries like auditwheel to embed shared libraries into wheels. So runtime system dependencies are included in wheel too. Mostly (see limitations https://github.com/pypa/auditwheel#limitations). So requirements shoud be somelike... * base_system_dependency: links to a OCI container, for example (open specifications if need to choose a technology). This mark minimal version of clib compatibility. * extra_base_system_repositories: We want modern libraries, with all patches, but running on old linux version to improve compatibility with clib. * build_dependencies: they must exists on base_system_dependencies or extra_base_system_repositories. ** build_system_dependencies: things used to test, compile and verify build. For example patchelf used by auditwheel ** build_python_dependencies: libraries used to build things. Like auditwheel used to pack into wheels runtime dependencies. * build_steps: Maybe even environment variables are a good idea if it will run in containers. It could include unit-tests, compilation, integration tests after compilation,... * artifacts: wheels, rpm packages, deb packages,... * runtime_dependencies: All things that can not be included in our generated artifacts. ** python_runtime_dependencies: The simple part ** system_runtime_dependencies: But for this section, system dependencies will not run containerized. Return to diffent names, available versions, incompatibilities,... I think that it's possible generate metadatas describing build requirements. Even if it fails for automated construction are a guide "TL;DR" of how install it. But if it can serve to generate automated constructions, repair metadatas will be easy and then automated construction will run. Then we'll need a pyproject2docker or pyproject2gitlabci or... Maybe this reference implementations should be needed to test specification. Metadatas should be used to avoid technological dependence in builds. If it no ends in a PEP is a good project to fill a private-repo with wheels. Regards, Javi
lasizoillo wrote:
Excellent thoughts, thanks for your input! The additional configuration fields you suggested capture valuable information and speak to a comprehensive build system that allows you to build in any image you want, with lots of customization. This is supported by PEP 600 -- Future 'manylinux' Platform Tags for Portable Linux Built Distributions (https://www.python.org/dev/peps/pep-0600/)...
This suggests that targeting the manylinux images will create the most value as it is assumed those images are what the majority of manylinux wheels will be built on. The work I am suggesting is to standardize "system set up" in manylinux containers prior to building. This would make the package author responsible for calling `wheel` to build, `auditwheel` to bundle and repair, and for uploading to their index of choice, so configuration for these actions is explicitly not captured in the proposed spec. In response to your suggestions for fields in the configuration: Fields that are appropriate to add: 1. `extra_system_repositories` 2. `steps` Fields that are not necessary: 1. `base_system`: This is not necessary when targeting only manylinux images as the section header can denote which image to run in and any "builder" can map between values, allowing custom urls doesn't create value here as there are only a few official manylinux images 2. `system_dependencies` and `build_system_dependencies`: why separate these into two groups? 3. `system_runtime_dependencies` I think this is a great idea but out of scope. This work should focus on build dependencies to avoid scope creep 4. `python_runtime_dependencies`: I believe this information is already captured in PEP-631: Dependency specification in pyproject.toml based on PEP 508 (https://www.python.org/dev/peps/pep-0631/) Updated working spec: ```toml [manylinux_build_spec.manylinux2014] extra_base_system_repositories: - FOO system_dependencies: - bar-1.0.0 environment_variables: - FOO=BAR python_dependencies: - foo==1.0.0 steps: - ./scripts/build_and_upload.sh --my_option ``` Thanks again! - Chris Antonellis
just one note:
3. `system_runtime_dependencies` I think this is a great idea but out of scope. This work should focus on build dependencies to avoid scope creep
It's not just scope creep. If a system (pip in this case) cannot provide those dependencies [*], then it really doesn't make sense to try to document / manage them here. In fact, the entire point of "manylinux" is that it provides a base system that is consistent -- requiring other run time dependencies negates that point. -CHB [*] in fact, the reason conda was developed, and is used, is because it CAN manage essentially any dependency. On Wed, Dec 23, 2020 at 10:49 AM Chris Antonellis < christopher.antonellis@gmail.com> wrote:
-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
lasizoillo wrote:
Excellent thoughts, thanks for your input! The additional configuration fields you suggested capture valuable information and speak to a comprehensive build system that allows you to build in any image you want, with lots of customization. This is supported by PEP 600 -- Future 'manylinux' Platform Tags for Portable Linux Built Distributions (https://www.python.org/dev/peps/pep-0600/)...
This suggests that targeting the manylinux images will create the most value as it is assumed those images are what the majority of manylinux wheels will be built on. The work I am suggesting is to standardize "system set up" in manylinux containers prior to building. This would make the package author responsible for calling `wheel` to build, `auditwheel` to bundle and repair, and for uploading to their index of choice, so configuration for these actions is explicitly not captured in the proposed spec. In response to your suggestions for fields in the configuration: Fields that are appropriate to add: 1. `extra_system_repositories` 2. `steps` Fields that are not necessary: 1. `base_system`: This is not necessary when targeting only manylinux images as the section header can denote which image to run in and any "builder" can map between values, allowing custom urls doesn't create value here as there are only a few official manylinux images 2. `system_dependencies` and `build_system_dependencies`: why separate these into two groups? 3. `system_runtime_dependencies` I think this is a great idea but out of scope. This work should focus on build dependencies to avoid scope creep 4. `python_runtime_dependencies`: I believe this information is already captured in PEP-631: Dependency specification in pyproject.toml based on PEP 508 (https://www.python.org/dev/peps/pep-0631/) Updated working spec: ```toml [manylinux_build_spec.manylinux2014] extra_base_system_repositories: - FOO system_dependencies: - bar-1.0.0 environment_variables: - FOO=BAR python_dependencies: - foo==1.0.0 steps: - ./scripts/build_and_upload.sh --my_option ``` Thanks again! - Chris Antonellis
just one note:
3. `system_runtime_dependencies` I think this is a great idea but out of scope. This work should focus on build dependencies to avoid scope creep
It's not just scope creep. If a system (pip in this case) cannot provide those dependencies [*], then it really doesn't make sense to try to document / manage them here. In fact, the entire point of "manylinux" is that it provides a base system that is consistent -- requiring other run time dependencies negates that point. -CHB [*] in fact, the reason conda was developed, and is used, is because it CAN manage essentially any dependency. On Wed, Dec 23, 2020 at 10:49 AM Chris Antonellis < christopher.antonellis@gmail.com> wrote:
-- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython
participants (3)
-
Chris Antonellis -
Christopher Barker -
lasizoillo