On May 6, 2016, at 10:59 PM, Nathaniel Smith wrote:

Here's that one-stop writeup/comparison of all the major configuration languages that I mentioned:

https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f

Thank you for this! A very nice summary. On Fri, May 6, 2016 at 11:14 PM, Donald Stufft wrote:

While I personally prefer YAML to any of the options on a purely syntax based level, when you weigh in all the other considerations for this I think that it makes sense to go with TOML for it.

I expect either YAML or TOML would be acceptable, based on this. I'll admit that I'd not heard of TOML before, but it warrants consideration (possibly for some of my projects as well). I've spent a fair bit of time using YAML with Ansible lately, as well as some time looking at RAML, and don't find myself worried about the syntax. Every oddness I've run across has been handled with an error when the content couldn't be parsed correctly, rather than unexpected behavior resulting from misunderstanding how it would be parsed. It's entirely possible I just haven't run across the particular problems Donald has run across, though. (The embedded Jinja2 in Ansible playbooks is another matter; let's not make that mistake.) -Fred -- Fred L. Drake, Jr. <fred at fdrake.net> "A storm broke loose in my mind." --Albert Einstein

On May 6, 2016, at 10:59 PM, Nathaniel Smith wrote:

...

Here's that one-stop writeup/comparison of all the major configuration languages that I mentioned:

https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f Thank you for this! A very nice summary.

On Fri, May 6, 2016 at 11:14 PM, Donald Stufft wrote:

...

While I personally prefer YAML to any of the options on a purely syntax

...

level, when you weigh in all the other considerations for this I think

I am +1 to TOML; it's INI (a human editable format) with data-types (I think it is even valid INI). I find the format pleasant to work with both in the available libraries and in the editor. -----Original Message----- From: Distutils-SIG [mailto:distutils-sig-bounces+tritium-list=sdamon.com@python.org] On Behalf Of Alex Grönholm Sent: Saturday, May 7, 2016 4:29 AM To: distutils-sig@python.org Subject: Re: [Distutils] comparison of configuration languages +1. I don't think the pathological cases of YAML syntax are of any concern in this context. Plus it has excellent tooling support, unlike TOML. 07.05.2016, 09:25, Fred Drake kirjoitti: based that it

...

makes sense to go with TOML for it. I expect either YAML or TOML would be acceptable, based on this. I'll admit that I'd not heard of TOML before, but it warrants consideration (possibly for some of my projects as well).

I've spent a fair bit of time using YAML with Ansible lately, as well as some time looking at RAML, and don't find myself worried about the syntax. Every oddness I've run across has been handled with an error when the content couldn't be parsed correctly, rather than unexpected behavior resulting from misunderstanding how it would be parsed. It's entirely possible I just haven't run across the particular problems Donald has run across, though.

(The embedded Jinja2 in Ansible playbooks is another matter; let's not make that mistake.)

-Fred

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

TOML-LD might work for representing JSONLD, as well. http://json-ld.org/#developers * https://github.com/RDFLib/rdflib-jsonld * https://github.com/digitalbazaar/pyld JSON-LD as a target makes sense because we're describing nodes (with attributes) and edges in a package graph. On Sat, May 7, 2016 at 7:06 AM, Wes Turner wrote:

+1 for YAML

YAML-LD (YAML & JSONLD) would make expressing the actual graphs for what could be "#PEP426JSONLD" much easier.

https://github.com/pypa/interoperability-peps/issues/31

On Saturday, May 7, 2016, Alex Grönholm wrote:

...

+1. I don't think the pathological cases of YAML syntax are of any concern in this context. Plus it has excellent tooling support, unlike TOML.

07.05.2016, 09:25, Fred Drake kirjoitti:

...

On May 6, 2016, at 10:59 PM, Nathaniel Smith wrote:

...

Here's that one-stop writeup/comparison of all the major configuration languages that I mentioned:

https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f

Thank you for this! A very nice summary.

On Fri, May 6, 2016 at 11:14 PM, Donald Stufft wrote:

...

While I personally prefer YAML to any of the options on a purely syntax based level, when you weigh in all the other considerations for this I think that it makes sense to go with TOML for it.

I expect either YAML or TOML would be acceptable, based on this. I'll admit that I'd not heard of TOML before, but it warrants consideration (possibly for some of my projects as well).

I've spent a fair bit of time using YAML with Ansible lately, as well as some time looking at RAML, and don't find myself worried about the syntax. Every oddness I've run across has been handled with an error when the content couldn't be parsed correctly, rather than unexpected behavior resulting from misunderstanding how it would be parsed. It's entirely possible I just haven't run across the particular problems Donald has run across, though.

(The embedded Jinja2 in Ansible playbooks is another matter; let's not make that mistake.)

-Fred

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On Sat, May 7, 2016 at 6:23 AM Paul Moore p.f.moore@gmail.com http://mailto:p.f.moore@gmail.com wrote: - YAML ought to be wonderful, but it ended up over-engineered (yes, we

can ignore the bits we don't care about). Also, pyYAML is a bit of an annoying dependency (big, reportedly slow unless you use the C version) - not something I'd want pip to have to vendor.

Perhaps a more limited YAML library like Poyo would address some of the concerns we have about using PyYAML. https://github.com/hackebrot/poyo#readme Poyo in particular may be *too* limited, but I wonder how much it would sway people if we used YAML but not PyYAML. Anyway, here are some relevant snippets from Poyo’s README: Please note that Poyo supports only a chosen subset of the YAML format. It can only read but not write and is not compatible with JSON. … Poyo is 100% Python and does not require any additional libs. Nick ​

On 8 May 2016 at 22:38, Nicholas Chammas wrote:

Perhaps a more limited YAML library like Poyo would address some of the concerns we have about using PyYAML.

I'm -1 on using "a subset of" a standard format. It immediately invites debate and confusion over *precisely* which subset is supported. I like (basic) YAML, but IMO it made the mistake of allowing itself to get too complex. Paul

On Sat, 7 May 2016 at 07:49 Nick Coghlan wrote:

On 7 May 2016 13:00, "Nathaniel Smith" wrote:

...

Here's that one-stop writeup/comparison of all the major configuration languages that I mentioned:

https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f

Thanks for that, and "yikes" on the comment handling variations in ConfigParser - you can tell I've never even tried to use end-of-line comments in INI files, and apparently neither has anyone I've worked with :)

Yeah, that's pretty bad. :/ I checked when ConfigParser was added to Python and it's late 1997: https://hg.python.org/cpython/rev/5b24cbb1f99b, so rather old and predates our stricter code quality rules for additions to the stdlib.

For YAML, my main concern isn't quirkiness of the syntax, or code quality in PyYAML, it's the ease with which you can expose yourself to security problems (even if *pip* loads the config file safely, that doesn't mean every other tool will). Since we don't need the extra power, the easiest way to reduce the collective attack surface is to use a strictly less powerful (but still sufficient) format.

For ast.literal_eval, we'd still need to come up with a way to do sections, key:value mappings and define rules for comments.

For completeness, I'll note that XML combines even more user unfriendly syntax than JSON with similar security risks to YAML.

So with the trade-offs laid out like that (and particularly the inconsistent comment and Unicode handling in ConfigParser), I'm prompted to favour following Rust in adopting TOML.

+1 for TOML from me as well. I know Paul brought up the lack of familiarity, but the format is simple and the Rust community is already fully dependent on it so at worst Rust + us could always just ignore future format versions if necessary. If TOML is the chosen format we could ask how long until a 1.0 release to know if we waited a month or so to implement we could make sure we're compliant with that version. I also checked pytoml at https://github.com/avakar/pytoml and it looks like it's pretty stable; no changes in the past 5 months except to support Python 3.5 and only 3 issues. And the format is simple enough that if someone had to fork the code like Nathaniel suggested or we did it from scratch it wouldn't be a huge burden. -Brett

Cheers, Nick.

P.S. I particularly like the idea of using extension sections to eventually consolidate other static config into a common file - that nicely addresses my concern with config file proliferation, since it opens the door to eventually subsuming other files like MANIFEST.in and setup.cfg as archiving and build systems are updated

...

-n

-- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Couple thoughts. Firstly, the human-editable bit: who in the last *decade* has been writing code using a non-syntax-aware/helping editor? Its a supremely uninteresting aspect IMO. On ConfigParser - yes, its horrid. OTOH we do get all the lines reliably, and setuptools will need to cover the unicode aspect itself in its own time. All we need to do is permit # inline as a comment - line a requirements.txt file for pip - and it becomes trivial to parse in all cases that we need *now*. Either we are defining the long term thing now, in which case that huge pile of complexity lands on us, and we have to get everything right. Or we are defining a thing which solves the present bug, and as long as we make sure it does not bind us in future, we're not hamstrung. E.g. use setup.cfg now. Add pybuild.toml later. (btw, terrible name, as pybuild is a thing in the debian space, and this will confuse the heck out of folk). https://wiki.debian.org/Python/Pybuild -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud

On May 7, 2016, at 5:05 PM, Robert Collins wrote:

Either we are defining the long term thing now, in which case that huge pile of complexity lands on us, and we have to get everything right.

Or we are defining a thing which solves the present bug, and as long as we make sure it does not bind us in future, we're not hamstrung.

E.g. use setup.cfg now. Add pybuild.toml later. (btw, terrible name, as pybuild is a thing in the debian space, and this will confuse the heck out of folk). https://wiki.debian.org/Python/Pybuild

I think this is roughly true, we could either do the simplest thing and just add ``setup_requires`` to ``setup.cfg`` and teach pip how to understand them and then worry about a new format later, or we can do a new format now and add a bit of complexity to what we need to specify (though I don't think _too_ much complexity, we don't have to define the build system stuff now, just make sure we don't back ourselves into a corner with that). I think either answer is OK, just the second one is a bit more work and we might either get the start of a better format _now_ or end up regretting what we pick when we add more things to it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Sat, May 7, 2016 at 6:04 PM, Brett Cannon wrote: For both options I hear "pick a new format", which suggests we might as

well do it from the get-go for clear separation of the new stuff and just bite the bullet instead of simply postponing a decision; it isn't like our format options are going to significantly change between now and later in the year.

Agreed. However, in another thread, I understood you to say that ALL we are talking about now is how to specify the build requirements. If that's the case, then we might a well as well just go with setup.cfg. However, I'd rather we were setting the stage for grater things -- in which case, let's go with a new config file. BTW, IIRC, there seemed to a consensus moving toward using a Python API, rather than a command line API for the mythical pluggable build systems.... In which case, we can require python, and could use python literals for configuration. With the discussion of PyYaml, I"m thinking more and more that something that can be parsed with only the stdlib is a good idea. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On 8 May 2016 at 08:46, Donald Stufft wrote:

...

On May 7, 2016, at 5:05 PM, Robert Collins wrote:

Either we are defining the long term thing now, in which case that huge pile of complexity lands on us, and we have to get everything right.

Or we are defining a thing which solves the present bug, and as long as we make sure it does not bind us in future, we're not hamstrung.

E.g. use setup.cfg now. Add pybuild.toml later. (btw, terrible name, as pybuild is a thing in the debian space, and this will confuse the heck out of folk). https://wiki.debian.org/Python/Pybuild

I think this is roughly true, we could either do the simplest thing and just add ``setup_requires`` to ``setup.cfg`` and teach pip how to understand them and then worry about a new format later, or we can do a new format now and add a bit of complexity to what we need to specify (though I don't think _too_ much complexity, we don't have to define the build system stuff now, just make sure we don't back ourselves into a corner with that).

I think either answer is OK, just the second one is a bit more work and we might either get the start of a better format _now_ or end up regretting what we pick when we add more things to it.

Hmm, are we perhaps forcing a false choice on ourselves here? 1. setup_requires in setup.py is the de facto baseline 2. d2to1 and pbr already use setup.cfg, so any arguments about the readability of the format are moot for users of those projects 3. we're not sure we're comfortable mandating the use of an ini-style format for all *future* Python projects So let's reduce our scope to: "We want *current users* of d2to1 and pbr to be able to declare those dependencies to pip and other installation tools in a way that avoids the implicit invocation of easy_install by setuptools" For *that* design goal, the conclusions I'd draw from Nathaniel's write-up would be those Łukasz suggested: - use https://pypi.python.org/pypi/configparser in Python 2.7 - explicitly define how to handle end-of-line comments We should perhaps also recommend that "#" be used for comments (even if the parser didn't enforce that), as I haven't seen anyone using ';' for comments since I stopped writing assembly code by hand. Improving the status quo for d2to1 and pbr users doesn't lock us into anything in terms of the "future of Python packaging as a whole". Individual build system authors may *choose* to take advantage of the setup.cfg mechanism, but they wouldn't be obliged to - they could keep using setup.py based bootstrapping if they preferred to do so. This approach would also provide the ability to iterate on a TOML based dependency declaration system *independent of pip itself*: setup.cfg could be used to bootstrap that system while it was still experimental, letting the idiosyncrasies get worked out before we commit to anything in pip itself. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Based on this email and Nathaniel's evaluation I've gone ahead and taken it upon myself to start writing a PEP so we have something concrete to work from. I'm hoping to have it done some time this week. On Sun, 8 May 2016 at 08:43 Donald Stufft wrote:

...

On May 8, 2016, at 9:13 AM, Nick Coghlan wrote:

So let's reduce our scope to: "We want *current users* of d2to1 and pbr to be able to declare those dependencies to pip and other installation tools in a way that avoids the implicit invocation of easy_install by setuptools"

I think it's less pbr and d2to1 that we're designing for here (although they'll benefit from it) but more like numpy.distutils. The reasoning is that pbr and such have already been able to architecture themselves so that they work within the current system. One of the goals here is to enable projects that haven't been able to (or can't reasonably) be written to work as a setuptools extension to be used here without requiring a manual pre-installation step.

The other side of this is anytime you have dependencies that aren't installed by pip (such as setup_requires being installed by setuptools) you end up with a confusing situation where settings don't get passed down into the "inner" installer (like ``--index-url``) or when they have different resolution algorithms (like setuptools supporting pre-releases by default) or when they support different formats (like pip not supporting eggs, but setuptools not supporting wheels).

However, while I think that a new format is more work just to make sure we don't back ourselves into a corner, I also don't think it's so much more work that it's really worth it to skip doing it now. Their isn't a whole lot of ways to solve the very narrow problem of "we need some dependencies installed prior to building". We need a field in a file that takes a list of PEP 508 style specifiers. We can skip any of the build system support right now and add it in later (and when we add it in, we can just make it so that a lack of a declaration is an implicit setuptool). So the only real things we need to decide on are:

1) What format to use? For this I think that it's been pretty clearly that TOML has had the greatest amount of support here and I think it represents the best set of trade offs for us.

2) What do we want to call this file? This one is pure bikeshed but also a bit important since this is one of the things that will be the hardest to change once we actually pick the name. Ideas I've seen so far (using the toml extension) are:

* pypa.toml - This one is specific to us which is good, but it's also a bit bad in that I don't think it's very widely known what the PyPA even is and I think that the PyPA works better if it's just sort of the thing in the background.

* pybuild.toml - This one might be a bit too oriented towards building rather than all of the possible uses of this file, but the bigger problem with it I think is that hte name clashes with pybuild on Debian which is their tool used for building Python packages.

* pip.toml - Also specific to us which is good, but I think it's a bit too overly specific maybe? One of our goals has been to make this stuff not dependent on a specific implementation so that we can replace it if we need to (as we more or less replaced distutils with setuptools, or easy_install with pip) however this also isn't exactly depending on an implementation-- just the name of a particular implementation. It does have the benefit that for a lot of people they associate "pip" with everything packaging related (e.g. I see people calling them "pip packages" now).

* pymeta.toml - This one might be reasonable, it's a bit generic but at least it has the ``py`` prefix to tie it to us. Not much to say about it otherwise.

Overall from those names, I think I probably like pymeta.toml the best (or maybe just ``meta.toml`` if we don't like the py prefix) but maybe other people have ideas/opinions that I haven't seen yet.

3) How do we structure the file and what keys do we use for just this part of the overall feature we're hoping to eventually get (and what semantics do we give these keys). We could just call it ``setup_requires``, but I think that's a bad name for something we're not planning on getting rid of since one of the eventual goals is to make ``setup.py`` itself optional. Another option is ``build_requires`` but that isn't quite right because we don't *just* need these dependencies for the build step (``setup.py bdist_wheel``) but also for the metadata step (``setup.py egg_info``). Although perhaps it doesn't really matter. Another option is to call them ``bootstrap_requires`` though it's a bit wonky to cram *build* requirements into that.

I guess if I were deciding this I would just call it build requirements because the nature of ``setup.py`` is that pretty much anything you need to execute ``setup.py`` is going to be a requirement for all ways you invoke ``setup.py`` (metadata and build and the legacyish direct to install). We could then say that if a project is missing this new file, then there is an implicit ``build_requires`` of ``["setuptools", "wheel"]`` but that as soon as someone defines this file that they are expected to accurately list all of their build dependencies in it.

Overall, my suggestion here would be to have a file called ``pymeta.toml`` (or ``meta.toml``) and have it look like::

[dependencies] build = [ "setuptools", "wheel>=1.0", ]

If at some point we decide we need to add a bootstrap_requires (and then the ability to add dynamic build requirements) we can do that by just saying that if you plan on having dynamic build requirements, you need to omit the build key under the [dependencies] section. This same thing could be used for other kinds of dependencies too (I've come around to the idea that you can't always declare a static set of dependencies in a sdist, but you can in a wheel) so that at some point in the future we could add additional keys like ``runtime`` to this where, if declared, will be assumed to be a static set of dependencies for that type of dependency.

This also doesn't prevent us from moving more of the metadata to this file in the future either. For an example, we could (at some point in the future, not now-- we shouldn't get bogged down in this now, it's just an example of this not tying us down too much) add something like::

[package] name = "mycoolpackage" version = "1.0"

[dependencies] build = ["setuptools", "wheel>=1.0"] runtime = ["requests==2.*"]

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On May 9, 2016, at 9:28 AM, Nick Coghlan wrote:

Looking at my previous ideas for semantic dependencies in PEP 426, what if we start in the near term by defining development requirements?

I think the biggest reason not to do this, but instead do something like build requirements is that development dependencies is already reasonably well addressed in a way that something other than setuptools can access it using extras. It's not as great as a dedicated key for it, but it works pretty OK. The thing that is really painful is setup_requires and how it forces you to delay importing until *during* the execution of the setup() function. We could try and lump setup_requires and development dependencies together, but that seems less than optimal to me. Unless someone's setup.py uses pytest, I'm not sure I see a reason for pytest to be installed anytime pip builds that project. A more concrete example would be pyca/cryptography, which has a development dependency that consists of 27MB of data which was purposely kept separate from cryptography itself so as not to incur an additional 27MB of download just to install cryptography. I think this *could* make sense, if we could reasonably assume that the 95% case would always be handled by wheels, but I don't think that we can. A lot of projects have compiled C code as part of them, and as soon as you do that you end up where installation from sdist is one of your main supported methods (yes Wheel will cover a lot of them, but only for the most popular platforms). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Mon, May 9, 2016 at 6:52 AM, Nick Coghlan wrote:

On 9 May 2016 at 23:38, Donald Stufft wrote:

...

...

On May 9, 2016, at 9:28 AM, Nick Coghlan wrote:

Looking at my previous ideas for semantic dependencies in PEP 426, what if we start in the near term by defining development requirements?

I think the biggest reason not to do this, but instead do something like build requirements is that development dependencies is already reasonably well addressed in a way that something other than setuptools can access it using extras. It's not as great as a dedicated key for it, but it works pretty OK. The thing that is really painful is setup_requires and how it forces you to delay importing until *during* the execution of the setup() function. We could try and lump setup_requires and development dependencies together, but that seems less than optimal to me. Unless someone's setup.py uses pytest, I'm not sure I see a reason for pytest to be installed anytime pip builds that project.

A more concrete example would be pyca/cryptography, which has a development dependency that consists of 27MB of data which was purposely kept separate from cryptography itself so as not to incur an additional 27MB of download just to install cryptography.

OK, that makes sense to me as a rationale for prioritising build dependencies over more general dev environment dependencies.

To feel confident we haven't overlooked a near term need by doing that, I think the main question I'd like to see the PEP explicitly cover is "when will these new requirements be implicitly installed?"

I currently believe the answers to that are:

1. When pip* is asked to install from an sdist 2. When pip* is asked to install from a VCS URL 3. When pip* is asked to install from a local directory 4. When pip* is asked to create a wheel file for any of those

* - or another compatible installer

If those answers are correct, then the metadata consumer will always be proceeding on to do a build in the use cases we currently care about, so it's OK that it may also be relying on those build requirements to generate metadata or create an sdist archive.

I think what we want is precisely, "what stuff do you need before you can invoke setup.py", and in the future that will be replaced with "what stuff do you need before you can invoke [our awesome hypothetical build system abstraction interface TBD, which will default to a wrapper around setup.py]". We all agree that there will be some sort of "build system" concept, and that invoking a build system is something that requires running arbitrary code, some of which might come from PyPI. It remains to be determined exactly what the boundary of that "build system" concept is (e.g., it definitely will have 'build a wheel'; not as obvious whether it will have 'build an sdist' or 'run tests'). But I think for now that's fine -- for now we can just say that eventually we will fill in the details of this interface, and the point of this part of the config file is that it lets you bootstrap into whatever that ends up being. (Notice that both the PEP 516/517 proposals actually distinguish between bootstrap-requires, which are what you need before invoking the build system, and build-requires, which is a dynamic hook for querying the build system to ask it what else it needs before it can build a wheel. Example of where the wheel-build requirements might vary at runtime: when building an extension that needs a C library, your build system might provide envvar that lets you pick between 'I'm building an RPM, so please just link to the system-provided copy of this library' versus 'I'm building a wheel, so please install the relevant pynativelib package -- making it a build requirement -- and then link to that'. I think we should focus narrowly on the bootstrapping problem right now, because it's not yet clear whether things like test dependencies actually make more sense as static metadata or as something you query the build system for dynamically. And a single static list of build-system-requirements is already enough to bandage over most of the really egregious setup_requires problems.) So I'd suggest: [build-system] requires = [ ... ] # Later we will add something like: # entry-point = ... -n -- Nathaniel J. Smith -- https://vorpus.org

"pymeta" feels very "inessentially weird" to me [1].

yeah... setup.toml ??? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

dstufft.toml imo Sent from my iPhone

On May 9, 2016, at 3:11 PM, Brett Cannon wrote:

You can all stop guessing at file names. The PEP will have a recommendation and you all can either agree or disagree at that point. Please don't give me more names to list in the rejected section. :)

On Sat, May 7, 2016 at 4:05 PM, Robert Collins wrote:

Couple thoughts.

Firstly, the human-editable bit: who in the last *decade* has been writing code using a non-syntax-aware/helping editor? Its a supremely uninteresting aspect IMO.

On ConfigParser - yes, its horrid. OTOH we do get all the lines reliably, and setuptools will need to cover the unicode aspect itself in its own time. All we need to do is permit # inline as a comment - line a requirements.txt file for pip - and it becomes trivial to parse in all cases that we need *now*.

ConfigObj is read/write with full Unicode support (and type validation) | Src: https://github.com/DiffSK/configobj | PyPI: https://pypi.python.org/pypi/configobj/5.0.6 | Docs: https://configobj.readthedocs.io/en/latest/ *Having read/write support avoids the primary issue (injection) with templating config files*

From https://github.com/DiffSK/configobj/blob/master/setup.py

LONG_DESCRIPTION = """**ConfigObj** is a simple but powerful config file reader and writer: an *ini file round tripper*. Its main feature is that it is very easy to use, with a straightforward programmer's interface and a simple syntax for config files. It has lots of other features though : * Nested sections (subsections), to any level * List values * Multiple line values * Full Unicode support * String interpolation (substitution) * Integrated with a powerful validation system - including automatic type checking/conversion - and allowing default values - repeated sections * All comments in the file are preserved * The order of keys/sections is preserved * Powerful ``unrepr`` mode for storing/retrieving Python data-types

Either we are defining the long term thing now, in which case that huge pile of complexity lands on us, and we have to get everything right.

Or we are defining a thing which solves the present bug, and as long as we make sure it does not bind us in future, we're not hamstrung.

E.g. use setup.cfg now. Add pybuild.toml later. (btw, terrible name, as pybuild is a thing in the debian space, and this will confuse the heck out of folk). https://wiki.debian.org/Python/Pybuild

-Rob

-- Robert Collins Distinguished Technologist HP Converged Cloud _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On 05/09/2016 12:30 PM, Barry Warsaw wrote:

Also, I think it makes a lot of sense to go with YAML even if it isn't the best most readable option. It's much more common than TOML so the learning curve will be lessened.

I'd rather learn some new syntax that is readable than be stuck with a pain in my eyes and in my brain. ;) -- ~Ethan~

On 05/09/2016 05:19 PM, Ethan Furman wrote:

On 05/07/2016 09:32 AM, Brett Cannon wrote:

...

I also checked pytoml at https://github.com/avakar/pytoml and it looks like it's pretty stable; no changes in the past 5 months except to support Python 3.5 and only 3 issues. And the format is simple enough that if someone had to fork the code like Nathaniel suggested or we did it from scratch it wouldn't be a huge burden.

After further consideration, and pytoml's author's comment about the spec changing without a version increase, I think we might be better off rolling our own. I like the general simplicity, and would stick with that, but I'd be a lot more comfortable if we had our spec that was more consistent. If there is interest I'll write the PEP and the tool. -- ~Ethan~

On 05/09/2016 08:35 PM, Nathaniel Smith wrote:

On Mon, May 9, 2016 at 7:22 PM, Ethan Furman wrote:

...

After further consideration, and pytoml's author's comment about the spec changing without a version increase, I think we might be better off rolling our own.

He's a bit confused -- they didn't change 0.4.0; they made changes in their dev branch working towards 1.0.0 (some cleanups related to the date/time stuff I think?"). But of course when you go to github it shows you the current dev version, and the dev version has a prominent link at the top to the 0.4.0 tag, so if you're skimming it's easy to misread it as saying that what you're looking at is 0.4.0.

Ah, okay. Thanks for clarifying! -- ~Ethan~

A few facts: * YAML is good enough for Salt, Ansible and numerous other common tools * The YAML standard has been stable for many years, unlike TOML which still hasn't even reached 1.0 * YAML has widespread tooling support, unlike TOML We all agree that JSON is not the solution. No comments, trailing commas etc. TOML isn't much better than ConfigParser in terms of representing nested structures. So far the ONLY objective problems with YAML seems to be the problematic implementation named PyYAML. If this is really the case, I'd gladly help build a better one just to prevent TOML from being chosen for this task. That we're even /considering/ building something as important as this on an unstable standard is pretty horrifying to me in itself. 10.05.2016, 06:37, Chris Barker kirjoitti:

Really?

writing Yet Another Markup Language (YAML :-) ) CAN'T be the simplest, best option.

...

After further consideration, and pytoml's author's comment about the spec changing without a version increase, I think we might be better off rolling our own.

I like the general simplicity, and would stick with that, but I'd be a lot more comfortable if we had our spec that was more consistent.

If we're going to do that, then why not the 'simple part of yaml'.

or Python literals. (if I recall, the main reason not to do that was that no other language has a lib to read it -- rolling out own does not solve that!)

Or just go with JSON -- I'm annoyed by it at times, but it's not SO bad.

(and you can kinda-sorta simulate comments with useless keys :-)

{ "comment": "this is just something i wanted to say here", ... }

or we could do "JSON with comments" -- not hard to write a tiny pre-processor before passing it off to the json lib.

Anyway -- let's avoid the temptation to role your own everything, and use something standard!

-CHB

--

Christopher Barker, Ph.D. Oceanographer

Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov mailto:Chris.Barker@noaa.gov

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

10.05.2016, 11:54, Paul Moore kirjoitti:

...

A few facts:

YAML is good enough for Salt, Ansible and numerous other common tools The YAML standard has been stable for many years, unlike TOML which still hasn't even reached 1.0 YAML has widespread tooling support, unlike TOML

We all agree that JSON is not the solution. No comments, trailing commas etc. TOML isn't much better than ConfigParser in terms of representing nested structures. Just as another data point, Cookiecutter started off using pyYAML, moved to ruamel.yaml, and ended up with their own (separated out into an independent project, but written specifically for Cookicutter) implementation of "some of" YAML, called poyo. I believe the reason

On 10 May 2016 at 08:38, Alex Grönholm wrote: they did this was repeated issues with the existing YAML libraries. And while the YAML-subset provided by poyo *looks* reasonable, I don't see any documentation of precisely *what* subset of YAML is implemented (apart from some examples). The poyo library is also very new, and I doubt it's seen much usage/testing outside of cookiecutter so far.

I would love to use YAML. I really would. But for pip, we need a robust, easy to vendor Python implementation (with no C dependency) that is safe (see http://community.embarcadero.com/blogs/entry/yaml-and-remote-code-execution-...). Writing our own is simply a way to end up with additional maintenance work, that we really don't have the resources for.

...

So far the ONLY objective problems with YAML seems to be the problematic implementation named PyYAML. If this is really the case, I'd gladly help build a better one just to prevent TOML from being chosen for this task. If you can get a robust, stable YAML library written fast enough to be an option for the PEP, then that would certainly be a possibility. But given that poyo has been under development for 5 months, are you going to be able to do better than that in a few weeks? (And see my comments above on poyo). Probably not all on my own, but if I get collaborators, then why not. The most important thing would be to have support for this so I know I'm not wasting my time. If PyYAML's test suite can be reused, that too would help a lot. Any new implementation would have to make sure they catch all the difficult corner cases. That we're even considering building something as important as this on an unstable standard is pretty horrifying to me in itself. Well, IMO, the state of things in terms of config file formats (and not just in Python) is itself pretty dreadful - every time I write an application, I am astounded that there are no good options for something as basic as a configuration file format.

Paul I've used PyYAML in safe mode with no problems, but the API is...awkward, to say the least. I'm not wondering why Donald is -1 on vendoring it to packaging tools.

In all this talk about using a YAML subset, I'm surprised no one mentioned YAMLish: https://pypi.python.org/pypi/yamlish It is a well defined subset of YAML and there are implementations in other programming languages. The problem with the 200+-lines-single-file library above is that it depends on PyYAML itself so, vendoring it will be challenging. Anyway, I think TOML 0.4.0 is good enough for our needs. On Tue, 10 May 2016 10:03 Nick Coghlan, wrote:

On 10 May 2016 at 18:54, Paul Moore wrote:

...

Well, IMO, the state of things in terms of config file formats (and not just in Python) is itself pretty dreadful - every time I write an application, I am astounded that there are no good options for something as basic as a configuration file format.

This is pretty normal for software - no good options, but a plethora of "good enough" ones. Hence https://xkcd.com/927/ :)

We just have a particularly exacting use case here, since we want:

- a format that's attractive for folks just learning to program in 2016 - a format that's attractive for folks that have been programming for 50+ years - a format that's easy to parse even in Python 2.6 - a format that's version control friendly - a format that's text editor friendly (syntax highlighting, etc)

For me, the two leading contenders out of the current discussion have been:

- the poyo subset of YAML 1.1 - TOML 0.4.0, as implemented by pytoml

The "subset of PyYAML" approach turns this into a documentation exercise (which subset?), and also runs into the problem that poyo doesn't handle serialisation yet, only *reading* YAML files.

For TOML, the main questions being asked are around how widely supported it is, and it's actually in a pretty good state on that front. 0.4.0 (rather than TOML in general) is stable by definition, and there are quite a few implementations of that across different languages: https://github.com/toml-lang/toml/blob/master/versions/en/toml-v0.4.0.md#imp...

The other big thing someone might want is a schema validator, and there I think it may be possible to just use jsonschema and validate the result of parsing the file, rather than validating the serialised form directly.

Cheers, Nick.

-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On Tue, May 10, 2016 at 1:54 AM, Paul Moore wrote:

I would love to use YAML. I really would. But for pip, we need a robust, easy to vendor Python implementation

conda has been using yaml forever (with pyyaml) , and whatever problem is has (and there are many), I don't think I've ever seen a single issue cause by yaml or pyyaml. Might be worth asking the conda devs. I think we're freaking out way too much about what *could* go wrong. Oh, and why not "JSON with comments and trailing commas" - it would be well defined and easy to implement. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Fri, 13 May 2016 at 10:47 Paul Moore wrote:

On 13 May 2016 at 18:15, Chris Barker wrote:

...

I think we're freaking out way too much about what *could* go wrong.

It's more that:pip would have to vendor pyyaml, and it's not small. I have no idea whether it's easy to vendor, either (does it have separate code paths for Python 2 and 3? I don't know, I've never looked). Also, we'd have to forego the C implementation - I have no idea how much of a performance hit that would give (of course, performance is hardly a key issue here) or how likely it is that bugs exist in the Python version that aren't normally noticed because people use the C implementation (which is definitely just speculation, conceded).

But I think the decision is made, so it's not worth debating any more, honestly.

No need to think; the decision is made and it's TOML. I know Chris doesn't mean to stir up trouble, but at this point if someone wants to propose something other than TOML they are going to have to write their own PEP.

On Fri, May 13, 2016 at 9:22 PM, Brett Cannon wrote:

No need to think; the decision is made and it's TOML. I know Chris doesn't mean to stir up trouble, but at this point if someone wants to propose something other than TOML they are going to have to write their own PEP.

​Not asking for any change but has anyone looked at libconfig http://www.hyperrealm.com/libconfig/libconfig_manual.html? ​It looks quite interesting: simple grammar and nesting support. What do you think of it? Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro

1. again, ConfigObj is pure Python, supports nesting, and is read/write (so there is no need to template injectable config files (e.g., with \n, =, #;etc. must be escaped)) | PyPI: https://pypi.python.org/pypi/configobj/5.0.6 | Src: https://github.com/DiffSK/configobj | Docs: https://configobj.readthedocs.io/en/latest/ 2. A grammar for configparser would make this more apparent https://hg.python.org/cpython/file/tip/Lib/configparser.py On Sunday, May 15, 2016, Robert Collins wrote:

On 15 May 2016 at 08:21, Ionel Cristian Mărieș javascript:;> wrote:

...

On Fri, May 13, 2016 at 9:22 PM, Brett Cannon

javascript:;> wrote:

...

...

No need to think; the decision is made and it's TOML. I know Chris

doesn't

...

mean to stir up trouble, but at this point if someone wants to propose something other than TOML they are going to have to write their own PEP.

Not asking for any change but has anyone looked at libconfig? It looks quite interesting: simple grammar and nesting support. What do you think of it?

I hadn't, but its certainly irrelevant here, as vendoring it suitably for pip would be excruciating due to the C dependency.

-Rob _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org javascript:; https://mail.python.org/mailman/listinfo/distutils-sig

​Not asking for any change but has anyone looked at libconfig http://www.hyperrealm.com/libconfig/libconfig_manual.html? ​It looks quite interesting: simple grammar and nesting support. What do you think of it As pointed out, it's a C lib. But as we all like writing tools, it wouldn't be very hard to write a Python parser for the format. But it's a bit C-y for my taste, and yet another configure language? Really? -CHB Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro

On Fri, May 13, 2016 at 10:15 AM, Chris Barker wrote:

On Tue, May 10, 2016 at 1:54 AM, Paul Moore wrote:

...

I would love to use YAML. I really would. But for pip, we need a robust, easy to vendor Python implementation

conda has been using yaml forever (with pyyaml) , and whatever problem is has (and there are many), I don't think I've ever seen a single issue cause by yaml or pyyaml. Might be worth asking the conda devs.

"vendor" here means "munge the library so that it becomes a pure-python subpackage of pip named pip._vendor.whatever". This is a specific technical challenge that conda doesn't attempt... -n -- Nathaniel J. Smith -- https://vorpus.org

On Tue, May 10, 2016 at 10:38 AM, Alex Grönholm wrote:

So far the ONLY objective problems with YAML seems to be the problematic implementation named PyYAML. If this is really the case, I'd gladly help build a better one just to prevent TOML from being chosen for this task. That we're even *considering* building something as important as this on an unstable standard is pretty horrifying to me in itself.

​Just my two cents here: every time, but every every time,​ I have to google around about how to create a multi-line string in YAML. There are too many ways to write the same thing. And lets not forget those damn sexagesimal literals. The complexity of that language is beyond repair, it's not a *library* problem. Just look at insanities like this http://stackoverflow.com/questions/3790454/in-yaml-how-do-i-break-a-string-o... ​or this http://blog.teamlazerbeez.com/2009/04/15/yaml-gotchas/​. Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro

For what it's worth, I've been following this thread, and I like the idea of using TOML for all the "pro" reasons posted so far. It's newness or not reaching 1.0 yet don't bother me as I believe the plans to specify TOML 0.4 or optionally support the later versions if they don't cause problems makes a lot of sense. The fact that the parser is 300+ lines of code and can be easily vendored is also a big plus. Given that rust is using TOML, if Python adopts it as well, that is big enough "market share" for me and people will get used to it soon enough. *Randy Syring* Husband | Father | Redeemed Sinner /"For what does it profit a man to gain the whole world and forfeit his soul?" (Mark 8:36 ESV)/ On 05/10/2016 03:38 AM, Alex Grönholm wrote:

A few facts:

* YAML is good enough for Salt, Ansible and numerous other common tools * The YAML standard has been stable for many years, unlike TOML which still hasn't even reached 1.0 * YAML has widespread tooling support, unlike TOML

We all agree that JSON is not the solution. No comments, trailing commas etc. TOML isn't much better than ConfigParser in terms of representing nested structures. So far the ONLY objective problems with YAML seems to be the problematic implementation named PyYAML. If this is really the case, I'd gladly help build a better one just to prevent TOML from being chosen for this task. That we're even /considering/ building something as important as this on an unstable standard is pretty horrifying to me in itself.

10.05.2016, 06:37, Chris Barker kirjoitti:

...

Really?

writing Yet Another Markup Language (YAML :-) ) CAN'T be the simplest, best option.

...

After further consideration, and pytoml's author's comment about the spec changing without a version increase, I think we might be better off rolling our own.

I like the general simplicity, and would stick with that, but I'd be a lot more comfortable if we had our spec that was more consistent.

If we're going to do that, then why not the 'simple part of yaml'.

or Python literals. (if I recall, the main reason not to do that was that no other language has a lib to read it -- rolling out own does not solve that!)

Or just go with JSON -- I'm annoyed by it at times, but it's not SO bad.

(and you can kinda-sorta simulate comments with useless keys :-)

{ "comment": "this is just something i wanted to say here", ... }

or we could do "JSON with comments" -- not hard to write a tiny pre-processor before passing it off to the json lib.

Anyway -- let's avoid the temptation to role your own everything, and use something standard!

-CHB

--

Christopher Barker, Ph.D. Oceanographer

Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov mailto:Chris.Barker@noaa.gov

_______________________________________________ Distutils-SIG maillist -Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On May 10, 2016, at 10:16 AM, Antoine Pitrou wrote:

On Tue, 10 May 2016 10:38:51 +0300 Alex Grönholm wrote:

...

TOML isn't much better than ConfigParser in terms of representing nested structures.

Indeed, that seems to be a strong point against TOML. If we don't care about nested structures that much, then ConfigParser should be more or less ok…

TOML is infinitely better at nested structured that ConfigParser, given that TOML actually *supports* nested structures beyond a level of 1. The only way to get anything like: [package.build] dependencies = ["setuptools", "wheel"] In ConfigParser is to add post-processing to the values, which then you're no longer a "ConfigParser" file, you're a "ConfigParser + Whatever random one off code you wrote to do post processing" file. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On May 10, 2016, at 10:30 AM, Antoine Pitrou wrote:

On Tue, 10 May 2016 10:24:10 -0400 Donald Stufft wrote:

...

TOML is infinitely better at nested structured that ConfigParser, given that TOML actually *supports* nested structures beyond a level of 1. The only way to get anything like:

[package.build] dependencies = ["setuptools", "wheel"]

In ConfigParser is to add post-processing to the values, which then you're no longer a "ConfigParser" file, you're a "ConfigParser + Whatever random one off code you wrote to do post processing" file.

The post-processing doesn't seem difficult enough to make any fuss about it, IMHO. The most important concern here should be how usable the format is for end users, not whether implementations need 20 additional lines of code to work with it.

I think TOML is more usable than ConfigParser and in particular I think that the adhoc post processing step makes ConfigParser inherently less usable because it forces a special syntax that is specific to this one file. It also means that there's no "right" answer for when you have two different implementations that interpret the same file differently. There's no spec to work off of so it ends up regressing to a smaller version of one of our current problems with the toolchain-- Everything is implementation defined by whatever the most popular tool at the time is.

(also, what is wrong with providing a pypa-specific library for parsing required configuration? are distlib / distil / pkgutil / the distutils-competitor-du-jour still alive?)

While we are providing unopinionated bindings like that for the PEPS, we don't want to make that "special" any more than it's just a reference implementation. Tools in languages other than Python will be expected to parse and read these files. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

10.05.2016, 18:00, Antoine Pitrou kirjoitti:

...

I think TOML is more usable than ConfigParser and in particular I think that the adhoc post processing step makes ConfigParser inherently less usable because it forces a special syntax that is specific to this one file. It also means that there's no "right" answer for when you have two different implementations that interpret the same file differently. That's true. OTOH, the question is how much better it is for users

On Tue, 10 May 2016 10:55:38 -0400 Donald Stufft wrote: that it's worthwhile bothering them with a syntax change that will require (at one point or another) migrating existing files. TOML doesn't seem that compelling to me in that regard (quite less than YAML, and I'm not a YAML fan).

(as an aside, if there's the question of forking an existing parser implementation for better vendorability, forking a YAML parser may be more useful to third-party folks than forking a TOML parser :-)) Amen to that, and that's exactly what I'd like to do. What should the parser be capable of to be accepted for this task? What are the requirements? Regards

Antoine. _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

10.05.2016, 18:14, Donald Stufft kirjoitti:

...

On May 10, 2016, at 11:00 AM, Antoine Pitrou wrote:

(as an aside, if there's the question of forking an existing parser implementation for better vendorability, forking a YAML parser may be more useful to third-party folks than forking a TOML parser :-))

I’m seeing what I can come up with (https://github.com/dstufft/yaml) though I don’t know that I feel like actually maintaining whatever it is I end up figuring out there. What exactly are your goals here?

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

10.05.2016, 18:26, Ethan Furman kirjoitti:

On 05/10/2016 08:14 AM, Donald Stufft wrote:

...

...

On May 10, 2016, at 11:00 AM, Antoine Pitrou wrote:

...

...

(as an aside, if there's the question of forking an existing parser implementation for better vendorability, forking a YAML parser may be more useful to third-party folks than forking a TOML parser :-))

I’m seeing what I can come up with (https://github.com/dstufft/yaml) though I don’t know that I feel like actually maintaining whatever it is I end up figuring out there.

Please no. I'd rather do xml than yaml.

Why do you hate it so much? I strongly prefer YAML to anything else I've seen here.

-- ~Ethan~

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

10.05.2016, 19:35, Ethan Furman kirjoitti:

On 05/10/2016 08:41 AM, Alex Grönholm wrote:

...

10.05.2016, 18:26, Ethan Furman kirjoitti:

...

...

Please no. I'd rather do xml than yaml.

Why do you hate it so much? I strongly prefer YAML to anything else I've seen here.

It's too complicated and error-prone. If we want buy-in from casual packagers then our configuration language needs to be simple to understand and simple to get right. (The amount of leading whitespace on a single line changes your data type? Really?? 0644 and 0x644 both map to 420? and 644 maps to 644?) Do you actually expect to use these in your project's configuration? No? Then why is this a problem for you in this case?

https://docs.saltstack.com/en/latest/topics/troubleshooting/yaml_idiosyncras...

https://ciaranm.wordpress.com/2009/03/01/yaml-sucks-gems-sucks-syck-sucks/

While YAML is more easily readable than XML, with XML you already know you're in hell so you tread more carefully. ;)

If we want to take the good ideas of YAML and make our own thing I'm okay with that -- but not YAML itself.

-- ~Ethan~ _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

On Tue, May 10, 2016 at 11:15 AM Donald Stufft donald@stufft.io http://mailto:donald@stufft.io wrote:

...

On May 10, 2016, at 11:00 AM, Antoine Pitrou wrote:

(as an aside, if there's the question of forking an existing parser implementation for better vendorability, forking a YAML parser may be more useful to third-party folks than forking a TOML parser :-))

I’m seeing what I can come up with (https://github.com/dstufft/yaml) though I don’t know that I feel like actually maintaining whatever it is I end up figuring out there.

Did you intend to fork from https://github.com/yaml/pyyaml? I believe the latest PyYAML is actually hosted on BitBucket: https://bitbucket.org/xi/pyyaml/ Also, a better starting point than PyYAML may be this existing fork of the library, ruamel.yaml: https://bitbucket.org/ruamel/yaml The fork makes several http://yaml.readthedocs.io/en/latest/pyyaml.html changes http://yaml.readthedocs.io/en/latest/detail.html#round-trip-including-commen... which may be relevant to you. Nick ​

On 10 May 2016 at 16:00, Antoine Pitrou wrote:

On Tue, 10 May 2016 10:55:38 -0400 Donald Stufft wrote:

...

I think TOML is more usable than ConfigParser and in particular I think that the adhoc post processing step makes ConfigParser inherently less usable because it forces a special syntax that is specific to this one file. It also means that there's no "right" answer for when you have two different implementations that interpret the same file differently.

That's true. OTOH, the question is how much better it is for users that it's worthwhile bothering them with a syntax change that will require (at one point or another) migrating existing files. TOML doesn't seem that compelling to me in that regard (quite less than YAML, and I'm not a YAML fan).

The one aspect that's missing from this discussion (largely because the PEP's still in the process of being written) is a clear statement of what capabilities we *need* from a config file format. I suspect we'll need: - Lists, for dependencies - Nesting (although this is more for "future expansion" by allowing us to namespace the keys) If we're looking to put package metadata in this file, we may also need multi-line string capabilities (for things like descriptions). Before we decide on a format based solely on tool support ("how easy is it for us to write the code?") we should probably see examples of the expected file config using the various different formats ("how easy is the format for users to use?"). Paul

On 11 May 2016 at 15:47, Greg Ewing wrote:

Having looked over the TOML spec, I like the simplicity of it (and I cringe from the complexity of YAML). The only thing I don't like about TOML is the way it cops out on nesting.

The only reason it does that as far as I can see is because of a dislike for significant indentation.

This is a feature, not a bug. The trick is that TOML allows *implicit* namespace creation when you only care about the innermost level: [this.is.nested] nice = "As a human reader & writer, you don't care about the nesting" [this-is-not-nested] nice = "As a human reader & writer, you don't care about the lack of nesting" The only folks that need to care about whether the configuration is actually nested or flat are those working with the config file *programmatically* - from the point of view of the folks hand editing the config file, it's always just the flat list of named key=value sections that they're used to in traditional ini files. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

07.05.2016, 17:48, Nick Coghlan kirjoitti:

On 7 May 2016 13:00, "Nathaniel Smith" mailto:njs@pobox.com> wrote:

...

Here's that one-stop writeup/comparison of all the major configuration languages that I mentioned:

https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f

Thanks for that, and "yikes" on the comment handling variations in ConfigParser - you can tell I've never even tried to use end-of-line comments in INI files, and apparently neither has anyone I've worked with :)

For YAML, my main concern isn't quirkiness of the syntax, or code quality in PyYAML, it's the ease with which you can expose yourself to security problems (even if *pip* loads the config file safely, that doesn't mean every other tool will). Since we don't need the extra power, the easiest way to reduce the collective attack surface is to use a strictly less powerful (but still sufficient) format.

Sounds like a far-fetched hypothetical problem. You're concerned about the custom tags provided by PyYAML? Do you happen to know a tool that defaults to loading files in unsafe mode?

For ast.literal_eval, we'd still need to come up with a way to do sections, key:value mappings and define rules for comments.

For completeness, I'll note that XML combines even more user unfriendly syntax than JSON with similar security risks to YAML.

So with the trade-offs laid out like that (and particularly the inconsistent comment and Unicode handling in ConfigParser), I'm prompted to favour following Rust in adopting TOML.

Cheers, Nick.

P.S. I particularly like the idea of using extension sections to eventually consolidate other static config into a common file - that nicely addresses my concern with config file proliferation, since it opens the door to eventually subsuming other files like MANIFEST.in and setup.cfg as archiving and build systems are updated

...

-n

-- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org

mailto:Distutils-SIG@python.org

...

https://mail.python.org/mailman/listinfo/distutils-sig

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

08.05.2016, 02:08, Donald Stufft kirjoitti:

...

On May 7, 2016, at 7:05 PM, Alex Grönholm mailto:alex.gronholm@nextday.fi> wrote:

07.05.2016, 17:48, Nick Coghlan kirjoitti:

...

On 7 May 2016 13:00, "Nathaniel Smith" wrote:

...

Here's that one-stop writeup/comparison of all the major configuration languages that I mentioned:

https://gist.github.com/njsmith/78f68204c5d969f8c8bc645ef77d4a8f

Thanks for that, and "yikes" on the comment handling variations in ConfigParser - you can tell I've never even tried to use end-of-line comments in INI files, and apparently neither has anyone I've worked with :)

For YAML, my main concern isn't quirkiness of the syntax, or code quality in PyYAML, it's the ease with which you can expose yourself to security problems (even if *pip* loads the config file safely, that doesn't mean every other tool will). Since we don't need the extra power, the easiest way to reduce the collective attack surface is to use a strictly less powerful (but still sufficient) format.

Sounds like a far-fetched hypothetical problem. You're concerned about the custom tags provided by PyYAML? Do you happen to know a tool that defaults to loading files in unsafe mode?

Yea, pyYAML itself does (yaml.load() does it unsafely, you have to use yaml.safe_load()).

I don’t think it’s that big of a deal though, we could easily add a thing to PyPI that rejects any YAML file that can’t be parsed in safe mode. The bigger deal to me is just that the library to work with it is a bit of a bear to use as a dependency.

Sounds like we'd need an alternate implementation of YAML then (I'd love to see a "yaml" module in the standard library too, but PyYAML isn't a good candidate for that, agreed).

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On 05/07/2016 04:11 PM, Robert Collins wrote:

Actually, Nathaniel didn't test vendorability of the libraries, and pip needs that. Pyyaml isn't in good shape there.

Um, what does "vendorability" mean? -- ~Ethan~

On May 09, 2016, at 08:30 PM, Donald Stufft wrote:

How hard is it to bundle it with pip by copying the source files into pip._vendor.*

Every time another package is vendored, a kitten falls off a unicorn. ;) Cheers, -Barry

Just so everyone knows, I'm ignoring this thread as the PEP I'm drafting with Donald and Nathaniel is nearly finished and thus has already settled on the file format discussion and I haven't heard a new point made on any file format proposal that has already been brought up previously. On Tue, 10 May 2016 at 10:10 Paul Moore wrote:

On 10 May 2016 at 13:40, Wolfgang wrote:

...

So why not use the ConfigParser available with Python and extend it to meet the requirements. Custom getters can be written and for the complex stuff ast.literal_eval() can be used to safely parse the complex list of requirements with comments.

Sadly, pip needs to work on Python 2.6+, so we can't use these new features of configparser.

Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig