[Distutils] Re: Distlib vs Packaging (Was: disable building wheel fora package)

On Fri., Sep. 21, 2018, 06:12 Tzu-ping Chung, wrote:

I agree with you about Pipfile. It is likely not something pip would not directly install packages based on. pip could potentially add a “lock” command that is able to generate a Pipfile.lock from Pipfile, or even start work in a fashion like npm etc., but conceptually, pip would only install things based on Pipfile.lock, and if it takes a Pipfile, it’s used to generate a Pipfile.lock (and maybe install from that).

Regarding the format of Pipfile.lock, the proposal of it being less tool-specific is interesting to me. Brett also touched on a similar proposition a while ago that maybe we could standardise a common lock file format (shared by Pipenv and Poetry in the context of the time), and I think it is a nice idea, too.

On the other hand, there are many other application dependency management tools out there, and as far as I know none of them actually have a lock file format with interoperability. JavaScript, for example, has maybe the most bipartisan state in that area (in npm and Yarn), and I don’t recall reading anything of this nature at all. I’m not saying this is wrong, but it’s interesting that Python, being relatively behind in this particular area, has this somewhat unique proposal here. (Again, this does not imply it’s either good or bad, just unique.)

So my motivation behind suggesting a standardized lockfile is my experience of having to deal with Node's two options. Either you end up alienating the subset of Node users who chose the other tool with the different lockfile or you have to put in extra effort to make sure to support both. I can give 2 concrete examples. One is I wrote a tool to generate the 3rd-party notices file for the Python extension for VS Code. Originally we were using Yarn, but then it turned out an internal tool at work only supported npm, so I had to choose to either not use the internal tool or switch us to npm (we did the latter because it turned out that package-lock.json is easier to work with for our use-case). Having a single format means tooling can be reused and this forcing function doesn't exist. The other example is serverless/PaaS cloud deployments. To support Python in that scenario you typically install dependencies upon deployment. So that means detecting and then using requirements.txt files on the cloud side. But then users ask for Pipfile.lock support. And the next request will be for pyporject.lock support for Poetry (and this exact scenario has happened to me at work when teams have asked for my opinion of what requirements file formats to support). Basically you are constantly trying to keep up with what the community is using which is tough to gauge. Having a single lock format means providers only need to support the one format and tools can come and go, innovating however they want as long as they produce the same artifact in the end (just like we have standardized on wheels while letting libraries have build tool flexibility through pyproject.toml). IOW I view libraries as having a build artifact of wheels while applications (should) have a build artifact of a lockfile. -Brett An extremely generic name like requirements.lock is probably not a good

idea, since it is not uncommon for a project to require multiple package managers (e.g. for multiple languages), and it would be a disaster if everyone uses generic names. If not tool-specific (e.g. yarn.lock), the name should at least be context-specific, like… I don’t know, pyproject? But that is taken :p (This is intentionally rhetoric to touch on the we-should-use-pyproject-for-this camp. To be clear: I am not in that camp, that’s likely a bad idea unless we rethink the whole application-library distinction Python packaging makes.)

TP

Sent from Mail https://go.microsoft.com/fwlink/?LinkId=550986 for Windows 10

*From: *Paul Moore *Sent: *21 September 2018 20:16 *To: *Nick Coghlan *Cc: *Michael Merickel ; Bert JW Regeer ; Distutils *Subject: *[Distutils] Re: Distlib vs Packaging (Was: disable building wheel fora package)

On Fri, 21 Sep 2018 at 11:41, Nick Coghlan wrote:

...

...

On Fri, 21 Sep 2018 at 05:47, Donald Stufft wrote:

...

...

On Sep 20, 2018, at 3:35 PM, Paul Moore wrote:

...

...

I don't think anyone's even spoken to the pip maintainers (yet?) about

...

...

supporting the pipfile format

...

...

...

...

That comes from me, I initially wrote the Pipfile as a proof of concept / sketch of an API for replacing the requirements.txt format, which Kenneth took and created pipenv from. At some point I plan on trying to push support for those ideas back into pip (not the virtual environment management bits though). That’s obviously my personal goal though, and doesn’t represent an agreed upon direction for pip.

...

...

And it's one where I think there are a couple of different levels of

...

support that are worth considering:

...

...

Q. Should pip support installing from Pipfile.lock files as well as

...

requirements.txt files?

...

...

A. Once the lock file format stabilises, absolutely, since this is

...

squarely in pip's "component installer" wheelhouse.

...

...

Q. Should "pip install" support saving the installed components to a

...

Pipfile, and then regenerating Pipfile.lock?

...

...

A. This is far less of a clearcut decision, as managing updates to a

...

file that's intended to be checked in to source control is where I

...

draw the line between "component installer" and

...

"application/environment dependency manager".

Speaking as a pip developer:

Where's there a good summary of the pipfile format, the pipfile.lock

format, and their relationship and contrast with requirements.txt? I

don't view https://github.com/pypa/pipfile as a "good summary",

because it explicitly states that pipfile is intended to *replace*

requirements.txt, and I disagree strongly with that.

Also, pipfile is human-readable, but pipfile.lock isn't. As far as I

know, pipfile.lock is currently generated solely by pipfile - before

pip consumes pipfile.lock, I'd like to see that format discussed and

agreed as a formal interop standard that any tools wanting to pass

data to pip (for the use case the standard describes) can use. One

obvious thing I'd like to consider is changing the name to something

less tool-specific - requirements.lock maybe?

As far as the pipfile format is concerned, I see that more as pipenv's

human readable input file that is used to *generate* the lock file,

and I don't see it as something pip should consume directly, as that

would mean pip overlapping in functionality with pipenv.

If I'm misunderstanding the relationship between pip and pipenv, or

between pipenv and pipfile, I'm happy to be corrected. But can I

suggest that the best way to do so would be to amend the project pages

that are giving me the impressions I have above, and pointing me at

the corrected versions? That way, we can make sure that any

misinformation is corrected at source...

Paul

PS Full disclosure - I've tried to use pipenv in a couple of local

projects, because of the hype about it being the "great new thing" and

found it basically of no use for my requirements/workflow. So I may

have a biased view of either pipenv, or how it's being presented. I'm

trying to be objective in the above, but my bias may have slipped

through.

--

Distutils-SIG mailing list -- distutils-sig@python.org

To unsubscribe send an email to distutils-sig-leave@python.org

https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/

Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/S...

-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/5...