support toml for pyproject support
Hi, I don't know if this was already debated but I don't know how to search in the whole archive of the list. For now the adoption of pyproject.toml file is more difficult because toml is not in the standard library. Each tool which wants to use pyproject.toml has to add a toml lib as a conditional or hard dependency. Since toml is now the standard configuration file format, it's strange the python does not support it in the stdlib lije it would have been strange to not have the configparser module. I know it's complicated to add more and more thing to the stdlib but I really think it is necessary for python packaging being more consistent. Maybe we could thought to a readonly lib to limit the added code. If it's conceivable, I'd be happy to help in it. Nice Day guys and girls. Jimmy
Hi Jimmy, and welcome, On Mon, Oct 08, 2018 at 09:10:40AM +0200, Jimmy Girardet wrote:
Hi,
I don't know if this was already debated but I don't know how to search in the whole archive of the list.
For now the adoption of pyproject.toml file is more difficult because toml is not in the standard library.
It is true that using third-party libraries is more difficult than using the std lib. That alone is not a reason to add a library to the std lib.
Each tool which wants to use pyproject.toml has to add a toml lib as a conditional or hard dependency.
Since toml is now the standard configuration file format,
It is? Did I miss the memo? Because I've never even heard of TOML before this very moment. Google Trends doesn't really support your assertion that TOML has become "the standard" for config files: # compare TOML, JSON and YAML https://trends.google.com/trends/explore?q=%2Fg%2F11c5zwr35t,%2Fm%2F05cntt,%... although it is trending upwards: https://trends.google.com/trends/explore?q=%2Fg%2F11c5zwr35t
it's strange the python does not support it in the stdlib lije it would have been strange to not have the configparser module.
We don't even ship a YAML library, and that seems to be far more popular than TOML. On the other hand, we do ship a plist library.
I know it's complicated to add more and more thing to the stdlib but I really think it is necessary for python packaging being more consistent.
Maybe we could thought to a readonly lib to limit the added code.
What is a readonly lib? -- Steve
On Mon, Oct 8, 2018 at 2:55 AM, Steven D'Aprano <steve@pearwood.info> wrote:
On Mon, Oct 08, 2018 at 09:10:40AM +0200, Jimmy Girardet wrote:
Each tool which wants to use pyproject.toml has to add a toml lib as a conditional or hard dependency.
Since toml is now the standard configuration file format,
It is? Did I miss the memo? Because I've never even heard of TOML before this very moment.
He's referring to PEPs 518 and 517 [1], which indeed standardize on TOML as a file format for Python package build metadata. I think moving anything into the stdlib would be premature though – TOML libraries are under active development, and the general trend in the packaging space has been to move things *out* of the stdlib (e.g. there's repeated rumblings about moving distutils out), because the stdlib release cycle doesn't work well for packaging infrastructure. -n [1] https://www.python.org/dev/peps/pep-0518/ https://www.python.org/dev/peps/pep-0517 -- Nathaniel J. Smith -- https://vorpus.org
On Mon, Oct 8, 2018 at 12:23 PM Nathaniel Smith <njs@pobox.com> wrote:
On Mon, Oct 8, 2018 at 2:55 AM, Steven D'Aprano <steve@pearwood.info> wrote:
On Mon, Oct 08, 2018 at 09:10:40AM +0200, Jimmy Girardet wrote:
Each tool which wants to use pyproject.toml has to add a toml lib as a conditional or hard dependency.
Since toml is now the standard configuration file format,
It is? Did I miss the memo? Because I've never even heard of TOML before this very moment.
He's referring to PEPs 518 and 517 [1], which indeed standardize on TOML as a file format for Python package build metadata.
I think moving anything into the stdlib would be premature though – TOML libraries are under active development, and the general trend in the packaging space has been to move things *out* of the stdlib (e.g. there's repeated rumblings about moving distutils out), because the stdlib release cycle doesn't work well for packaging infrastructure.
If I had the energy to argue it I would also argue against using TOML in those PEPs. I personally don't especially care for TOML and what's "obvious" to Tom is not at all obvious to me. I'd rather just stick with YAML or perhaps something even simpler than either one.
He's referring to PEPs 518 and 517 [1], which indeed standardize on TOML as a file format for Python package build metadata.
I think moving anything into the stdlib would be premature though – TOML libraries are under active development, and the general trend in the packaging space has been to move things *out* of the stdlib (e.g. there's repeated rumblings about moving distutils out), because the stdlib release cycle doesn't work well for packaging infrastructure.
If I had the energy to argue it I would also argue against using TOML in those PEPs. I personally don't especially care for TOML and what's "obvious" to Tom is not at all obvious to me. I'd rather just stick with YAML or perhaps something even simpler than either one.
This thread isn't about regretting past decisions but what makes sense given current realities though. / Anders
I agree here. I briefly urged against using the less used TOML format, but I have no real skin in the game around packaging. I like YAML, but that's also not in the standard library, even if more widely used. But given that packaging is committed to TOML, I think that's a strong case for including a library in stdlib. The PEP 517/518 authors had their reasons that were accepted. Now there is broad ecosystem that is built on that choice. Let's support it. On Mon, Oct 8, 2018, 8:03 AM Anders Hovmöller <boxed@killingar.net> wrote:
He's referring to PEPs 518 and 517 [1], which indeed standardize on TOML as a file format for Python package build metadata.
I think moving anything into the stdlib would be premature though – TOML libraries are under active development, and the general trend in the packaging space has been to move things *out* of the stdlib (e.g. there's repeated rumblings about moving distutils out), because the stdlib release cycle doesn't work well for packaging infrastructure.
If I had the energy to argue it I would also argue against using TOML in those PEPs. I personally don't especially care for TOML and what's "obvious" to Tom is not at all obvious to me. I'd rather just stick with YAML or perhaps something even simpler than either one.
This thread isn't about regretting past decisions but what makes sense given current realities though.
/ Anders
On Mon, Oct 8, 2018 at 12:49 PM Jimmy Girardet <ijkl@netc.fr> wrote:
Hi,
Hi Jimmy and welcome! :)
I don't know if this was already debated but I don't know how to search in the whole archive of the list.
For now the adoption of pyproject.toml file is more difficult because toml is not in the standard library.
Each tool which wants to use pyproject.toml has to add a toml lib as a conditional or hard dependency.
Since toml is now the standard configuration file format, it's strange the python does not support it in the stdlib lije it would have been strange to not have the configparser module.
Let's wait till TOML hits 1.0 before adding it to the standard library. It's still at 0.5 right now. I am personally in favor of adding a standard library module for TOML, after it hits 1.0 and there's some stability after the release.
I know it's complicated to add more and more thing to the stdlib but I really think it is necessary for python packaging being more consistent.
TOML has a fairly unambiguous specification so I don't think the choice of library should really affect what data gets loaded. If there are differences across implementations, due to the TOML specification unintentionally being ambiguous, please do file an issue on GitHub. :)
Maybe we could thought to a readonly lib to limit the added code.
I don't think that would be as helpful as possibly a round-tripping parser-writer combination but I'll refrain from pushing for that *right now*.
If it's conceivable, I'd be happy to help in it.
Nice Day guys and girls.
Jimmy
Cheers, Pradyun (pip maintainer, TOML Core member)
_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
On Mon, Oct 8, 2018 at 4:53 AM Erik Bray <erik.m.bray@gmail.com> wrote:
If I had the energy to argue it I would also argue against using TOML in those PEPs. I personally don't especially care for TOML and what's "obvious" to Tom is not at all obvious to me. I'd rather just stick with YAML or perhaps something even simpler than either one.
I feel the same way. (Somebody was requesting extensive TOML support for mypy and was also waving those PEPs in front of us.) -- --Guido van Rossum (python.org/~guido)
If I had the energy to argue it I would also argue against using TOML
in those PEPs.
I partook in that discussion, and I still have no idea why TOML was chosen, over, say, a defined subset of YAML, or a slightly extended JSON. But the folks that were highly invested and putting the work in made a choice, so here we are. But if it’s in the PEPs, it seems time to define a version used ( 1.0 would be good, but often that’s actually pretty arbitrary) and get an implementation into the stdlib. If the TOML folks don’t think it’s stable enough for that, I’ve got to wonder if it was a good choice! We’re have enough trouble with really slow adoption of the new ways of doing packaging, not even providing the tools seems to really defeat the purpose. -CHB
On Mon, 8 Oct 2018 09:26:12 -0400 David Mertz <mertz@gnosis.cx> wrote:
I agree here. I briefly urged against using the less used TOML format, but I have no real skin in the game around packaging. I like YAML, but that's also not in the standard library, even if more widely used.
Agreed with David. Also, please note that one argument against YAML is its complexity, which logically also entails implementation complexity (and therefore makes it less likely that a YAML implementation would enter the stdlib). Regards Antoine.
Hello everyone, it has been almost a year after this issue was brought up. I suggest starting discussion on this again. I created an issue in python/peps repo https://github.com/python/peps/issues/1133
Hi, I am sure this was suggested somewhere before, but I couldn't find any references. I think toml file parser should be part of standard library. I know there is discussions like https://discuss.python.org/t/pep-594-removing-dead-batteries-from-the-standa.... But toml is now official format for build system via pyproject.toml. So it is very odd that we have to use external tools to parse it. Also many tools like black already started using it as configuration file. I know that pyproject.toml wasn't intended for such usecase, but nonetheless it very well may be a centralised configuration file for many QA and dev tools like flake8, coverage etc.
but was pointed that discussion should start here. In the era of 0.x software https://0ver.org I think it is not feasible to always wait till some standard, library or software becomes 1.x, take flask for example it has been years of usage in production before it got 1.x release. TOML standard pretty well defined, it was already chosen as fileformat for our new build system requirements https://www.python.org/dev/peps/pep-0518/. Standard library evolves and grows. I don't see a problem of having not TOML parser in standard library. Or at least some official package that will be defacto shipped with python itself as battery on installation. As @Anders Hovmöller said in this thread https://mail.python.org/archives/list/python-ideas@python.org/message/OEKD6M...
This thread isn't about regretting past decisions but what makes sense given current realities though.
I see this could be solved in one of these ways 1) We add TOML in standard library and have versioned parsers like TOMLParser05, TOMLParser06 etc and current version would be set TOMLParser = TOMLParser05, this way there won't be a problem when standard library and next version of python will ship new standard. Deprecation of parser classes could follow our standard deprecation procedure 2) There could be official TOML parser package under new https://github.com/psf group supervision, which will could be installed to platforms during python installation as pip does currently
On Fri, Jul 26, 2019 at 5:15 AM Sardorbek Imomaliev < sardorbek.imomaliev@gmail.com> wrote:
Hello everyone, it has been almost a year after this issue was brought up. I suggest starting discussion on this again. I created an issue in python/peps repo https://github.com/python/peps/issues/1133
Hi, I am sure this was suggested somewhere before, but I couldn't find any references. I think toml file parser should be part of standard library. I know there is discussions like https://discuss.python.org/t/pep-594-removing-dead-batteries-from-the-standa.... But toml is now official format for build system via pyproject.toml. So it is very odd that we have to use external tools to parse it.
It's actually not that odd. Think about all the setuptools usage and that isn't in the stdlib either.
Also many tools like black already started using it as configuration file. I know that pyproject.toml wasn't intended for such usecase, but nonetheless it very well may be a centralised configuration file for many QA and dev tools like flake8, coverage etc.
We relaxed the intent of pyproject.toml for those use-cases, so it's officially considered okay. :)
but was pointed that discussion should start here. In the era of 0.x software https://0ver.org I think it is not feasible to always wait till some standard, library or software becomes 1.x, take flask for example it has been years of usage in production before it got 1.x release. TOML standard pretty well defined, it was already chosen as fileformat for our new build system requirements https://www.python.org/dev/peps/pep-0518/. Standard library evolves and grows. I don't see a problem of having not TOML parser in standard library. Or at least some official package that will be defacto shipped with python itself as battery on installation.
1.0 actually isn't that far off: https://github.com/toml-lang/toml/projects/1, so I would still much rather wait for that rather than rush ahead, especially since we have survived this long without a TOML parser in the stdlib. -Brett
As @Anders Hovmöller said in this thread https://mail.python.org/archives/list/python-ideas@python.org/message/OEKD6M...
This thread isn't about regretting past decisions but what makes sense given current realities though.
I see this could be solved in one of these ways 1) We add TOML in standard library and have versioned parsers like TOMLParser05, TOMLParser06 etc and current version would be set TOMLParser = TOMLParser05, this way there won't be a problem when standard library and next version of python will ship new standard. Deprecation of parser classes could follow our standard deprecation procedure 2) There could be official TOML parser package under new https://github.com/psf group supervision, which will could be installed to platforms during python installation as pip does currently _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/VU6LRL... Code of Conduct: http://python.org/psf/codeofconduct/
On Jul 26, 2019, at 04:57, Sardorbek Imomaliev <sardorbek.imomaliev@gmail.com> wrote:
In the era of 0.x software https://0ver.org I think it is not feasible to always wait till some standard, library or software becomes 1.x,
I think _in general_ waiting for 1.0 makes sense. Many projects don’t make any compatibility promises until 1.0 (think YAML 0.9). But different projects use version numbers differently. And TOML says:
As of version 0.5.0, TOML should be considered extremely stable. The goal is for version 1.0.0 to be backwards compatible (as much as humanly possible) with version 0.5.0. All implementations are strongly encouraged to become 0.5.0 compatible so that the transition to 1.0.0 will be simple when that happens.
And that’s been the official word unchanged for over a year now, so they seem serious about this.
Standard library evolves and grows.
But it evolves and grows slowly—not only on an 18-month cycle, but with strict rules about backward compatibility, and arguments about any substantial improvements. Would you be happy using today’s TOML library, with nothing but minor fixes, in 3 years? If not, you shouldn’t want it in the stdlib. Lots of libraries that lots of people need still need to evolve rapidly or independently. A big reason for working so hard on the pip ecosystem is so things like requests or numpy can be there for millions of users without having to get pulled into the stdlib to die. And, even if the packaging tools need it, and can freeze on an existing version and a specific subset of functionality, that still isn’t necessarily a reason to pull it in. It could be treated like requests: setuptools/pip/ensurepip bundle a version of requests internally, while end users can install a newer (or a specific) version in site-packages so they aren’t burdened by the release cycle. Would that not be appropriate for the way packaging needs TOML?
Thanks, Brett. You've brought good points. And I wasn't necessarily advocating about adding TOML support to stdlib, actually it was more about having a defacto library for TOML parsing. Currently setuptools, pip and others(now even requests) are under supervision(not total but some) of PSF. I suggest doing same for TOML parsing. If we are bringing new standard to python ecosystem we should provide all parts needed for this tool/lib/standard to be usable "out of the box" for most users.
Hi, Adnrew! I agree with you for the most parts. But one thing to note my actual point was not about specifically adding TOML parsing lib to stdlib, but about having some defacto TOML parser under PSF supervision. So it could be used by most users without any additional jumps through hoops if they wan't to add some pypackage.toml support/parsing for their tooling
On Mon, Jul 29, 2019 at 12:57 AM Sardorbek Imomaliev < sardorbek.imomaliev@gmail.com> wrote:
Thanks, Brett. You've brought good points. And I wasn't necessarily advocating about adding TOML support to stdlib, actually it was more about having a defacto library for TOML parsing. Currently setuptools, pip and others(now even requests) are under supervision(not total but some) of PSF. I suggest doing same for TOML parsing. If we are bringing new standard to python ecosystem we should provide all parts needed for this tool/lib/standard to be usable "out of the box" for most users.
That's unfortunately not a general thing we can make happen. Someone needs to implement a TOML package and then ask to move it to under the PSF after gaining enough traction. Your best option would be to ask one of the TOML package authors if they are interested in moving their project.
participants (13)
-
Anders Hovmöller
-
Andrew Barnert
-
Antoine Pitrou
-
Brett Cannon
-
Chris Barker - NOAA Federal
-
David Mertz
-
Erik Bray
-
Guido van Rossum
-
Jimmy Girardet
-
Nathaniel Smith
-
Pradyun Gedam
-
Sardorbek Imomaliev
-
Steven D'Aprano