Vote to adopt extensionlib
https://github.com/ofek/extensionlib is the library based on https://hackmd.io/@gaborbernat/py-packaging-summit-2022#Building-C-Extension... The goal is to iterate on it while implementing a PoC for an existing project like Mypyc or Cython, then write the PEP to standardize the extension module builder interface, registration, and configuration. I'll start the vote with an explicit +1
I'll second it. On Mon, Jun 13, 2022 at 4:43 AM Ofek Lev <ofekmeister@gmail.com> wrote:
https://github.com/ofek/extensionlib is the library based on https://hackmd.io/@gaborbernat/py-packaging-summit-2022#Building-C-Extension...
The goal is to iterate on it while implementing a PoC for an existing project like Mypyc or Cython, then write the PEP to standardize the extension module builder interface, registration, and configuration.
I'll start the vote with an explicit +1 _______________________________________________ PyPA-Committers mailing list -- pypa-committers@python.org To unsubscribe send an email to pypa-committers-leave@python.org https://mail.python.org/mailman3/lists/pypa-committers.python.org/ Member address: gaborjbernat@gmail.com
-1 -- I think this is too soon. Has there been any discussion about this outside of PyCon US? If there has, I'd appreciate a link -- https://discuss.python.org/search?q=extensionlib shows nothing. If there hasn't been any such discussion, I'd like to flag that we *should* discuss this, in a non-PyCon setting, before heading into a vote. Otherwise, folks who didn't attend PyCon are operating with significantly less context. Beyond that, I'd like to see this get adopted in multiple backends and have some functionality before we adopt this into the PyPA. It's a cool pre-PEP implementation and we don't typically have a PEP be tied to an implementation so... let's not be "too fast" with moving things into the PyPA umbrella. Best, Pradyun On Mon, Jun 13, 2022 at 4:43 AM Ofek Lev <ofekmeister@gmail.com> wrote:
https://github.com/ofek/extensionlib is the library based on https://hackmd.io/@gaborbernat/py-packaging-summit-2022#Building-C-Extension...
The goal is to iterate on it while implementing a PoC for an existing project like Mypyc or Cython, then write the PEP to standardize the extension module builder interface, registration, and configuration.
I'll start the vote with an explicit +1 _______________________________________________ PyPA-Committers mailing list -- pypa-committers@python.org To unsubscribe send an email to pypa-committers-leave@python.org https://mail.python.org/mailman3/lists/pypa-committers.python.org/ Member address: pradyunsg@gmail.com
I'm not sure why the reluctance. This project is actively used by a PyPA project (hatch), so feels like OK to also adopt it. Note adoption under the PyPA umbrella does not imply either approval stamp or completeness. Also, Ofek could host this project also under the hatch repository and we'd not have a vote over that (note the hatch Github Repository already hosts both the hatch and hatchling PyPI projects). Feels odd to allow maintaining a new dependency under the existing Github Repository, but then require maturity and wide adoption for hosting on its own. This just encourages monolithic repositories. On Mon, Jun 13, 2022 at 10:02 AM Pradyun Gedam <pradyunsg@gmail.com> wrote:
-1 -- I think this is too soon.
Has there been any discussion about this outside of PyCon US? If there has, I'd appreciate a link -- https://discuss.python.org/search?q=extensionlib shows nothing. If there hasn't been any such discussion, I'd like to flag that we *should* discuss this, in a non-PyCon setting, before heading into a vote. Otherwise, folks who didn't attend PyCon are operating with significantly less context.
Beyond that, I'd like to see this get adopted in multiple backends and have some functionality before we adopt this into the PyPA. It's a cool pre-PEP implementation and we don't typically have a PEP be tied to an implementation so... let's not be "too fast" with moving things into the PyPA umbrella.
Best, Pradyun
On Mon, Jun 13, 2022 at 4:43 AM Ofek Lev <ofekmeister@gmail.com> wrote:
https://github.com/ofek/extensionlib is the library based on https://hackmd.io/@gaborbernat/py-packaging-summit-2022#Building-C-Extension...
The goal is to iterate on it while implementing a PoC for an existing project like Mypyc or Cython, then write the PEP to standardize the extension module builder interface, registration, and configuration.
I'll start the vote with an explicit +1 _______________________________________________ PyPA-Committers mailing list -- pypa-committers@python.org To unsubscribe send an email to pypa-committers-leave@python.org https://mail.python.org/mailman3/lists/pypa-committers.python.org/ Member address: pradyunsg@gmail.com
_______________________________________________ PyPA-Committers mailing list -- pypa-committers@python.org To unsubscribe send an email to pypa-committers-leave@python.org https://mail.python.org/mailman3/lists/pypa-committers.python.org/ Member address: gaborjbernat@gmail.com
On Mon, 13 Jun 2022, at 10:09, Bernat Gabor wrote:
I'm not sure why the reluctance. This project is actively used by a PyPA project (hatch), so feels like OK to also adopt it.
Is it? The search https://github.com/pypa/hatch/search?q=extensionlib shows no hits. Maybe it's a rename of something within hatch? The extensionlib docs also encourage extending the [project] table in pyproject.toml (https://ofek.dev/extensionlib/config/ ), something that's not meant to happen without a PEP. So I'm reading this as an experimental, proof of concept, thing, not a library backends should be adopting as documented straight away.
Note adoption under the PyPA umbrella does not imply either approval stamp or completeness.
Officially it may not mean that, but in practice I think it looks very much like a stamp of approval to people outside PyPA. I'm not casting a vote at the moment, but my initial reaction is that it looks like an experiment, and I'd like to see such experiments living outside the PyPA umbrella. Thomas
-1. I agree with Pradyun. I think it's too soon to adopt this under the PyPA, as anyone who wasn't at PyCon has no idea what this is about, and no way of finding out to inform their vote. Also, if it needs a PEP to standardise the interface, I think it would be reasonable to expect that we agree the PEP before making this a PyPA project. Regarding Layday's point, I'm not aware of any project that was created directly as a PyPA project, and in all honesty, I don't think we should do that. Being under the PyPA umbrella doesn't make *that* much difference to a project, but one of the main differences is the perception of a certain level of "official" status (for better or worse), and I don't think a new, experimental project, still under rapid development, is a good fit for that sort of status. As regards whether this "encourages monolithic repositories", I don't think that's a real concern. I hope that all PyPA members have a sense of responsibility and wouldn't simply add new functionality to an existing project as a way of avoiding discussion on whether the code should be under the PyPA umbrella. And conversely, I'd assume that a proposal to split off *mature* functionality that's part of an existing project into its own library would be supported. So I think we're fine as regards ensuring existing projects remain modular. But I don't think that applies to extensionlib, which (as far as I can tell) is new, experimental functionality at this point. Paul On Mon, 13 Jun 2022 at 10:02, Pradyun Gedam <pradyunsg@gmail.com> wrote:
-1 -- I think this is too soon.
Has there been any discussion about this outside of PyCon US? If there has, I'd appreciate a link -- https://discuss.python.org/search?q=extensionlib shows nothing. If there hasn't been any such discussion, I'd like to flag that we *should* discuss this, in a non-PyCon setting, before heading into a vote. Otherwise, folks who didn't attend PyCon are operating with significantly less context.
Beyond that, I'd like to see this get adopted in multiple backends and have some functionality before we adopt this into the PyPA. It's a cool pre-PEP implementation and we don't typically have a PEP be tied to an implementation so... let's not be "too fast" with moving things into the PyPA umbrella.
Best, Pradyun
On Mon, Jun 13, 2022 at 4:43 AM Ofek Lev <ofekmeister@gmail.com> wrote:
https://github.com/ofek/extensionlib is the library based on https://hackmd.io/@gaborbernat/py-packaging-summit-2022#Building-C-Extension...
The goal is to iterate on it while implementing a PoC for an existing project like Mypyc or Cython, then write the PEP to standardize the extension module builder interface, registration, and configuration.
I'll start the vote with an explicit +1 _______________________________________________ PyPA-Committers mailing list -- pypa-committers@python.org To unsubscribe send an email to pypa-committers-leave@python.org https://mail.python.org/mailman3/lists/pypa-committers.python.org/ Member address: pradyunsg@gmail.com
_______________________________________________ PyPA-Committers mailing list -- pypa-committers@python.org To unsubscribe send an email to pypa-committers-leave@python.org https://mail.python.org/mailman3/lists/pypa-committers.python.org/ Member address: p.f.moore@gmail.com
I don't have an opinion on extensionlib, but I do believe that all of the PyPI related repositories like warehouse, conveyor, trove-classifiers, linehaul, etc were all created directly as PyPA projects. PyPI is kind of unique though in that the PyPI admins/devs get to universally decide by fiat what projects make up PyPI, which isn't true for projects for end users. I will note that PEP 609 does call out "organically creating projects within the PyPA" as an option under the "Determine which projects should be under the Guidance of the PyPA", and the guidelines on https://www.pypa.io/en/latest/members/ pretty much give PyPA members fairly wide latitude to adopt whatever they think is right. All that to say that while I think it's fine to say that we don't think this project warrants inclusion yet, that I wouldn't want to introduce any strict rules, these sorts of things are best left to human judgement :) On 6/13/2022 5:54:59 AM, Paul Moore <p.f.moore@gmail.com> wrote: -1. I agree with Pradyun. I think it's too soon to adopt this under the PyPA, as anyone who wasn't at PyCon has no idea what this is about, and no way of finding out to inform their vote. Also, if it needs a PEP to standardise the interface, I think it would be reasonable to expect that we agree the PEP before making this a PyPA project. Regarding Layday's point, I'm not aware of any project that was created directly as a PyPA project, and in all honesty, I don't think we should do that. Being under the PyPA umbrella doesn't make *that* much difference to a project, but one of the main differences is the perception of a certain level of "official" status (for better or worse), and I don't think a new, experimental project, still under rapid development, is a good fit for that sort of status. As regards whether this "encourages monolithic repositories", I don't think that's a real concern. I hope that all PyPA members have a sense of responsibility and wouldn't simply add new functionality to an existing project as a way of avoiding discussion on whether the code should be under the PyPA umbrella. And conversely, I'd assume that a proposal to split off *mature* functionality that's part of an existing project into its own library would be supported. So I think we're fine as regards ensuring existing projects remain modular. But I don't think that applies to extensionlib, which (as far as I can tell) is new, experimental functionality at this point. Paul On Mon, 13 Jun 2022 at 10:02, Pradyun Gedam wrote:
-1 -- I think this is too soon.
Has there been any discussion about this outside of PyCon US? If there has, I'd appreciate a link -- https://discuss.python.org/search?q=extensionlib shows nothing. If there hasn't been any such discussion, I'd like to flag that we *should* discuss this, in a non-PyCon setting, before heading into a vote. Otherwise, folks who didn't attend PyCon are operating with significantly less context.
Beyond that, I'd like to see this get adopted in multiple backends and have some functionality before we adopt this into the PyPA. It's a cool pre-PEP implementation and we don't typically have a PEP be tied to an implementation so... let's not be "too fast" with moving things into the PyPA umbrella.
Best, Pradyun
On Mon, Jun 13, 2022 at 4:43 AM Ofek Lev wrote:
https://github.com/ofek/extensionlib is the library based on https://hackmd.io/@gaborbernat/py-packaging-summit-2022#Building-C-Extension...
The goal is to iterate on it while implementing a PoC for an existing project like Mypyc or Cython, then write the PEP to standardize the extension module builder interface, registration, and configuration.
I'll start the vote with an explicit +1 _______________________________________________ PyPA-Committers mailing list -- pypa-committers@python.org To unsubscribe send an email to pypa-committers-leave@python.org https://mail.python.org/mailman3/lists/pypa-committers.python.org/ Member address: pradyunsg@gmail.com
_______________________________________________ PyPA-Committers mailing list -- pypa-committers@python.org To unsubscribe send an email to pypa-committers-leave@python.org https://mail.python.org/mailman3/lists/pypa-committers.python.org/ Member address: p.f.moore@gmail.com
_______________________________________________ PyPA-Committers mailing list -- pypa-committers@python.org To unsubscribe send an email to pypa-committers-leave@python.org https://mail.python.org/mailman3/lists/pypa-committers.python.org/ Member address: donald@stufft.io [d23be46e-e946-4702-8982-e966390fc72c]
enscons https://github.com/dholth/cryptacular/blob/master/SConstruct#L42 also builds extension modules. It uses distutils to get the (correct?) compiler arguments https://github.com/dholth/enscons/blob/master/enscons/cpyext.py but it invokes the compiler independently of distutils. How do meson, maturin, extensionlib get the correct compiler arguments for building extension modules?
I read through extensionlib. It seems like it wants to get lists of build inputs and outputs, and then participate as a hybrid build system even implementing the clean() command in the caller. It doesn't appear to have anything to do with C compiler arguments or the specifics of building. SCons (not specifically enscons which is a way to use SCons to build wheels) expects to do the build, and then tell you what happened. It spends enough time building a dependency graph of the source code that you don't really want to do that an extra time just to say what you would have done. enscons is intentionally completely different than setuptools; the conceptual mismatch shows when trying to match the two systems - for example if you expected a https://docs.python.org/3/distutils/apiref.html#distutils.core.Extension . SCons could also build anything, like generated Python code in pysdl2-cffi, not just extension modules, and it only rebuilds when the dependencies have changed. If it had to participated in another build system, a two function "build all extension modules (tell me what you did)" call, and a "clean" call would be a better match. On Wed, Jun 15, 2022, at 8:04 AM, Daniel Holth wrote:
enscons https://github.com/dholth/cryptacular/blob/master/SConstruct#L42 also builds extension modules. It uses distutils to get the (correct?) compiler arguments https://github.com/dholth/enscons/blob/master/enscons/cpyext.py but it invokes the compiler independently of distutils.
How do meson, maturin, extensionlib get the correct compiler arguments for building extension modules? _______________________________________________ PyPA-Committers mailing list -- pypa-committers@python.org To unsubscribe send an email to pypa-committers-leave@python.org https://mail.python.org/mailman3/lists/pypa-committers.python.org/ Member address: dholth@fastmail.fm
We've got two mechanisms to incorporate projects under PyPA: the first is to create a new project directly under the PyPA organisation and the second is to move an established project. I would like to hear if the first option has ever been exercised and what criteria were applied in the past. I think it would make more sense to frame this as a proposal to create a new project under PyPA even if the repo does already exist outside the PyPA org. In both cases, I think it'd be good: * At the very least, to have a high-level overview of what the library does and how it works. A code sample would also help. * To have the involvement of other interested parties - some uptake. In the former case: * If some investigation were done upfront to validate the premise of the project. In the latter case: * If the library were used to build extensions for one or several packages to identify and resolve oftentimes fundamental issues inherent to a new concept and implementation. Best D Sent with Proton Mail secure email. ------- Original Message ------- On Monday, June 13th, 2022 at 6:43 AM, Ofek Lev <ofekmeister@gmail.com> wrote:
https://github.com/ofek/extensionlib is the library based on https://hackmd.io/@gaborbernat/py-packaging-summit-2022#Building-C-Extension...
The goal is to iterate on it while implementing a PoC for an existing project like Mypyc or Cython, then write the PEP to standardize the extension module builder interface, registration, and configuration.
I'll start the vote with an explicit +1 _______________________________________________ PyPA-Committers mailing list -- pypa-committers@python.org To unsubscribe send an email to pypa-committers-leave@python.org https://mail.python.org/mailman3/lists/pypa-committers.python.org/ Member address: layday@protonmail.com
high-level overview of what the library does and how it works
While the PEP will mandate a class protocol, this library provides an easy to use ABC builders can inherit if they wish that will also have fundamental utilities that all builders would likely use e.g. sysconfig things. The library also provides a runner that backends can use, if they wish, that invokes all configured builders. I document everything here: https://ofek.dev/extensionlib/
To have the involvement of other interested parties
Yes this is planned, and is definitely a requirement before the PEP.
participants (8)
-
Bernat Gabor -
Daniel Holth -
Donald Stufft -
layday -
Ofek Lev -
Paul Moore -
Pradyun Gedam -
Thomas Kluyver