On Wed., 26 Sep. 2018, 2:40 pm Chris Jerdonek, <chris.jerdonek@gmail.com> wrote:

My advice to you (and which I've been following myself) would be for
you to break changes into small, tightly-focused PR's, and to have
only one outstanding at a time (or perhaps two if non-overlapping).
The main reason is that there aren't too many people reviewing and
merging changes (mainly just Pradyun in my case), so I don't want to
overwhelm them. I've been trying to follow the maxim "go slow to go
fast." Also, plan to expect / be okay with a fair of amount of
preliminary refactoring PR's before splitting something out might even
be possible. I.e. you will probably need a lot of "set up," especially
if it's a gnarlier area of the code base.

Note that the difficulty of doing this is a key part of why my own recommendation to the folks working on refactoring pipenv's internals was to handle the API design process reasonably independently of pip (at least initially): the point of providing a public Python level API is to address the cases that pip itself doesn't address, and while refactoring pip to use the same common set of libraries is important to reducing the long term collective maintenance burden, it's unlikely to be the quickest way of addressing the near term ecosystem level API need for projects like pip-tools, pipenv, and poetry. 

So if anyone's irritated by that approach, best to focus your irritation on me for providing that advice, not on the folks doing the work of breaking out pipenv's internals as reusable libraries that can hopefully eventually be shared with other projects (including pip) as their APIs stabilise. When they say that's the advice they received, it's an accurate statement, it's just that by participating directly in distutils-sig rather than proxying through me, they're now also hearing from folks that would have favoured a different approach :)

Cheers,

Nick.

P.S. At a larger meta-level, I find it's worth keeping in mind that there will be always be ebbs and flows of convergence and divergence in any large scale organisational system, and I really like http://theagilepirate.net/archives/1392 as a write-up of that in a corporate context.

In an open source community context, I think the same phenomenon shows up as cycles of competition (different approaches to superficially common problems with no clear winner amongst them) and consolidation (improved understanding of the problem space resulting in clearer articulation of the design trade-offs involved, and the emergence of "obvious" default choices at particular points in the problem space).

--Chris


>
> Dan Ryan // pipenv maintainer
> gh: @techalchemy
>
>> On Sep 25, 2018, at 5:47 PM, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
>>
>>> On Tue, Sep 25, 2018 at 6:46 AM, Dan Ryan <dan@danryan.co> wrote:
>>> As far as the pipenv roadmap, we are rewriting a good chunk of it but it essentially the atomic things we do or would like to do with regards to shared functionality at a high level:
>>>
>>> - parse user input/requirements
>>> - retrieve package metadata
>>> - resolve dependencies (not in pip right now obviously) — may involve building wheels/sdists
>>> - download / build packages
>>> - install packages (into a virtualenv sometimes)
>>> - uninstall packages (from a virtualenv sometimes)
>>> - list packages etc (in a virtualenv sometimes)
>>>
>>> We know how these are accomplished in pip and we know how we are handling them both in pipenv and in our alternative implementations, which I believe still have some dependency on pip. For us the roadmap is basically to figure out
>>>
>>> - where do the abstractions live
>>> - how do we start
>>> - ???
>>
>> Thanks, Dan. I think Donald answered this in an email much earlier in
>> this thread -- the one he sent on Sept. 19 beginning:
>>
>> """
>> My general recommendation if you want a Python
>> implementation/interface for something pip does, is:
>> - Open an issue on the pip repository to document your intent and to
>> make sure that there is nobody there who is against having that
>> functionality split out. ...
>> ...
>> """
>>
>> In addition, to ensure you're taking the approach of splitting out
>> parts of pip instead of creating yet another implementation separate
>> from pip (leading to duplicated work as I said), I would recommend
>> also doing the following:
>>
>> File PR's in pip to refactor out various functionality that makes
>> sense into stand-alone bits with their own tests (e.g. unit tests,
>> etc). This would live in pip but be disentangled from the rest of pip
>> from a code dependency perspective. It could even be in a separate
>> "future libraries" subdirectory if the pip committers are okay with
>> it. (Optional: since you're already vendoring pip, you could perhaps
>> use this factored out code at your own risk, like you're already doing
>> with some of pip's code base anyway.)
>>
>> The above is something I started doing at a small scale -- not to
>> break it out into a library, but for its own sake because it makes it
>> easier to maintain and work on pip as a code base, fixing bugs, etc.
>>
>> Some of the nice things about this approach are that--
>>
>> 1. It can be done even before the steps that Donald outlined (like I'm
>> doing), e.g. so that you don't need to wait for a PEP to be finished.
>> And it could even help in the later creation of such PEPs.
>> 2. By doing so, as a side effect of the PR process, it automatically
>> causes communication with pip maintainers and works towards the
>> "splitting out parts of pip" goal -- all without duplicating any work
>> because both pip and pipenv developers are aware and collaborating on
>> these changes.
>> 3. It is incremental as you go, ensuring that it is compatible with
>> pip and that pip is using it, etc.
>> 4. It improves pip in the process (as well as pipenv, if you're still
>> vendoring in some form). In other words, the work is useful from the
>> beginning, whether or not it's ever broken out into a separately
>> maintained library.
>> 5. It is forward progress towards the goal of a separate library.
>>
>> --Chris
>>
>>
>>
>>>
>>> Dan Ryan // pipenv maintainer
>>> gh: @techalchemy
>>>
>>>>> On Sep 25, 2018, at 8:11 AM, Paul Moore <p.f.moore@gmail.com> wrote:
>>>>>
>>>>>> On Tue, 25 Sep 2018 at 12:53, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
>>>>>>
>>>>>> On Tue, Sep 25, 2018 at 3:47 AM, Tzu-ping Chung <uranusjr@gmail.com> wrote:
>>>>>> We are using pip internals for things pip wasn’t implemented for. Specifically,
>>>>>> Pipenv uses pip’s package-fetching functions to implement its platform-agnostic
>>>>>> resolver. pip does not have this, so there’s no functional overlap here. Those
>>>>>> utilities are used to build something that doesn’t exist in pip, so there’s no
>>>>>> duplicated efforts.
>>>>>>
>>>>>> My recent focus on making sense of packaging implementations and splitting out
>>>>>> parts of pip is exactly to prevent potential duplicated efforts. If we can’t use pip
>>>>>> internals, let’s make things we want to use *not* internal!
>>>>>
>>>>> I don't see this practice of "splitting out parts of pip" actually
>>>>> being followed so far though, and I want to tell you how it's leading
>>>>> to duplicated efforts right now -- even for the PR's I happen to be
>>>>> working on today.
>>>>
>>>> [...]
>>>>
>>>>> So by creating implementations of functions separate from pip instead
>>>>> of splitting them out of pip, it's leading to duplicated work for
>>>>> people working on pip's code base. A process of splitting something
>>>>> out of pip would I think involve a process more like what I'm
>>>>> following -- namely to refactor functionality in pip *first* so it can
>>>>> be used separately, and then pull that out so you only have one
>>>>> implementation at any point in time. Otherwise, you wind up creating
>>>>> two versions of similar functionality that not only need to be created
>>>>> twice, but also later reconciled if the plan is merge them back
>>>>> together.
>>>>
>>>> This is a very good point, and thanks for the concrete example. For
>>>> the record, I'm also a little annoyed that pipenv have been doing all
>>>> this work in isolation. There's been very little communication between
>>>> the pip and pipenv projects, and I think that's contributed to the
>>>> problem. I, for one, wasn't even aware for a long time that they were
>>>> embedding their own copy of pip. When we made pip's internals in
>>>> private in pip 10, we got essentially no feedback from anyone saying
>>>> that it would be a problem, and finding out that it had a sufficiently
>>>> major impact on pipenv that they had to embed the old version of pip
>>>> *after* the release was (to put it politely) suboptimal :-(
>>>>
>>>> Having said all this, this thread is the start of an attempt to work
>>>> better together and I think we should be glad that it's happening and
>>>> try to make it work. But I don't think we've got off to a particularly
>>>> good start and it will need work. In particular, I'd like to see an
>>>> attempt from *both* projects to clarify our goals - I've been speaking
>>>> solely for myself so far, it would be good to get the other pip
>>>> developers' views and set out a sort of roadmap, and for pipenv to do
>>>> the same.
>>>>
>>>> Paul
>>>> --
>>>> Distutils-SIG mailing list -- distutils-sig@python.org
>>>> To unsubscribe send an email to distutils-sig-leave@python.org
>>>> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
>>>> Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/OGXUNDRBLTNPOWLLFBVF45SW4SRGYSPP/
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-leave@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/JQGH4HPIKWXGEWYEARESL5KC5WYEPR2S/