Dear Eric: Is Distutils2 kind of your baby? Simply, you are the only one working on it! I am curious as to what you mean by this statement, when I get some spare round tuits?

From the bug tracker page, this has been around for over a year! I applied the diff file that was attached to the page to the current Python-3.3.0a2, and it doesn't make a bit of difference on the outcome...

I also added the path to sys.path, like this: sys.path.append(os.getcwd()) and it doesn't make a difference! After the upcoming Distutils2 sprint, when will the work be patched into Distutils2? When will we see the benefit of all the work that will be done??? 1) http://dl.dropbox.com/u/47247655/create_setup_cfg.py 2) http://dl.dropbox.com/u/47247655/setup_custom.py 3) http://dl.dropbox.com/u/47247655/setup.cfg 1) Will create the setup.cfg file if one is not there...? The rest, you know about already... I have updated 2 and 3 and uploaded the new ones... Sincerely yours, Rob G. Healey On Fri, Apr 20, 2012 at 5:32 PM, Éric Araujo wrote:

Good evening,

...

BTW, these two files are within the same directory within the same project! Here is the error that I get: [...] PackagingModuleError: setup_custom

This looks like http://bugs.python.org/issue11637 The project directory needs to be added to sys.path when running command hooks.  I have a patch in progress and will commit it when I get some spare round tuits.

Thank you for testing and sending feedback!

Regards _______________________________________________ Distutils-SIG maillist  -  Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

-- Sincerely yours, Rob G. Healey

Dear Eric: On Fri, Apr 20, 2012 at 10:51 PM, Éric Araujo wrote:

Hi again,

...

Is Distutils2 kind of your baby?  Simply, you are the only one working on it!

It’s more like my foster kid.  At first there was distutils, in the standard library since 1999.  After a few years it stopped being actively developed, and people started to rely on undocumented behavior and internal things.  Four years ago Tarek Ziadé became the new maintainer for distutils and attacked the big pile of bugs and feature requests.  He hit a barrier when his changes broke other people’s code, so it was decided two years ago to stop adding new features or clean up code in distutils and instead start anew in a project named distutils2, where backward compatibility would not stall all improvement efforts. That’s when I started contributing, as a Google Summer of Code student and then a volunteer.  Tarek focused on distutils2 and passed on the maintenance of distutils to me, and later I also became the maintainer of distutils2, i.e. I reply to most bugs and make the releases.  Tarek is still the lead of the project and weighs in for important changes.

I understand now! My role at gramps is something like that too in one manner of updating the two plugins in the webreport directory, NarrativeWeb.py and WebCal.py... I am full owner and maintainer of my plugin, EditExifImageMetadata.py, which allows editing and creation of metadata fields for images... Now anyone else can also modify this plugin and also the plugins listed above, but it mainly falls on my shoulders...

...

I am curious as to what you mean by this statement, when I get some spare round tuits?

That’s a wordplay from the expression “I’ll get around to it” → “I’ll get a round tuit”.

I like it...

...

From the bug tracker page, this has been around for over a year! Bugs live from a few minutes to a few years; with more than 300 bugs on my list, I can’t get to everything at the same time.  For the last months I’ve also been busy with paperwork and a new job.  But be sure the bug is not forgotten; it’s one of the things that will be fixed before distutils2 1.0 and Python 3.3.0.

After the sprint and you release Distutils2-1.0a5, would you also update the version_info too? If you import distutils2, and then distutils2.version_info, you get a3 not a4?

...

I applied the diff file that was attached to the page to the current Python-3.3.0a2, and it doesn't make a bit of difference on the outcome...

Obviously, I didn't get that far in reading the bug tracker! I will re-download Python-3.3.0a2 and re-install it without the extra diff file...

If you read the full bug report you’ll see that this patch was already applied and only fixes setup hooks; I said that I was working on a patch for commands hooks but I did not upload it as it was not finished.

I am more than willing to do testing of the patch for you if you would like? Is there a way for me to get svn or git access to the developmental version of Distutils2? Since I am legally blind and have kidney failure, I spend a whole lot of time at home on my computer! It is also one extra thing that I do at gramps is alpha and beta testing...

...

I also added the path to sys.path, like this: sys.path.append(os.getcwd()) and it doesn't make a difference!

What would I do in Linux to add my directory to the permanent $PYTHONPATH???

Probably because you did that in your code, but that makes no difference because it happens after pysetup is started and tries to import your module.  I don’t know if I’ll have time to fix this bug tomorrow, as I’ll have mentoring to do, so no promises here.

Is there access to the bug tracker for simply Distutils2 only? I would like to see it and start getting familiar with the Distutils2 code???

...

After the upcoming Distutils2 sprint, when will the work be patched into Distutils2?  When will we see the benefit of all the work that will be done???

The changes from the sprint will go into the main repository immediately, and after a short while they will get ported into packaging in the cpython repository.  I want to release distutils2 1.0a5 soon after the sprint.

I will certainly be looking forward to it! Once again, I have a lot of time on my hands, so I am willing to do alpha/ beta testing...

...

1) http://dl.dropbox.com/u/47247655/create_setup_cfg.py [...] 1) Will create the setup.cfg file if one is not there...? As you’re using open('setup.cfg', 'w'), the file will be recreated each time you run the script.  I’m not sure you need that script; why not just create the config file once and keep it updated when you have changes?

You are correct about, why not just update the setup.cfg file instead of creating one each time I make changes! Sincerely yours, Rob G. Healey

Regards _______________________________________________ Distutils-SIG maillist  -  Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

-- Sincerely yours, Rob G. Healey

On 04/21/2012 01:51 AM, Éric Araujo wrote:

Hi again,

...

Is Distutils2 kind of your baby? Simply, you are the only one working on it! It’s more like my foster kid. At first there was distutils, in the standard library since 1999. After a few years it stopped being actively developed, and people started to rely on undocumented behavior and internal things. Four years ago Tarek Ziadé became the new maintainer for distutils and attacked the big pile of bugs and feature requests. He hit a barrier when his changes broke other people’s code, so it was decided two years ago to stop adding new features or clean up code in distutils and instead start anew in a project named distutils2, where backward compatibility would not stall all improvement efforts. That’s when I started contributing, as a Google Summer of Code student and then a volunteer. Tarek focused on distutils2 and passed on the maintenance of distutils to me, and later I also became the maintainer of distutils2, i.e. I reply to most bugs and make the releases. Tarek is still the lead of the project and weighs in for important changes.

Georg Brandl was trying to herd cats on outstanding PEPs today: http://mail.python.org/pipermail/python-dev/2012-May/119157.html Is there a PEP for the "packaging" package? Is there any sort of unfinished business I can help with? - C

...

I am curious as to what you mean by this statement, when I get some spare round tuits? That’s a wordplay from the expression “I’ll get around to it” → “I’ll get a round tuit”.

...

From the bug tracker page, this has been around for over a year! Bugs live from a few minutes to a few years; with more than 300 bugs on my list, I can’t get to everything at the same time. For the last months I’ve also been busy with paperwork and a new job. But be sure the bug is not forgotten; it’s one of the things that will be fixed before distutils2 1.0 and Python 3.3.0.

...

I applied the diff file that was attached to the page to the current Python-3.3.0a2, and it doesn't make a bit of difference on the outcome... If you read the full bug report you’ll see that this patch was already applied and only fixes setup hooks; I said that I was working on a patch for commands hooks but I did not upload it as it was not finished.

...

I also added the path to sys.path, like this: sys.path.append(os.getcwd()) and it doesn't make a difference! Probably because you did that in your code, but that makes no difference because it happens after pysetup is started and tries to import your module. I don’t know if I’ll have time to fix this bug tomorrow, as I’ll have mentoring to do, so no promises here.

...

After the upcoming Distutils2 sprint, when will the work be patched into Distutils2? When will we see the benefit of all the work that will be done??? The changes from the sprint will go into the main repository immediately, and after a short while they will get ported into packaging in the cpython repository. I want to release distutils2 1.0a5 soon after the sprint.

...

1) http://dl.dropbox.com/u/47247655/create_setup_cfg.py [...] 1) Will create the setup.cfg file if one is not there...? As you’re using open('setup.cfg', 'w'), the file will be recreated each time you run the script. I’m not sure you need that script; why not just create the config file once and keep it updated when you have changes?

Regards _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

On 05/01/2012 02:28 PM, Paul Moore wrote:

On 1 May 2012 17:40, Chris McDonough wrote:

...

Georg Brandl was trying to herd cats on outstanding PEPs today:

http://mail.python.org/pipermail/python-dev/2012-May/119157.html

Is there a PEP for the "packaging" package? Is there any sort of unfinished business I can help with?

AFAIK, there's no specific PEP for packaging (there are a number of related PEPs, but nothing specific like a roadmap).

I'm sure Éric can give you much better pointers on what would be useful, but one issue I've tried to raise a few times, and more recently Jim Fulton raised here (http://mail.python.org/pipermail/distutils-sig/2012-March/018323.html) is that of binary distribution support in packaging2. I've never had the time to shepherd a proposal through beyond the "initial debate" stage, and I know it's not getting high on Éric's list of priorities, but it would be good to see some movement on this.

I can offer opinions, Windows testing, and assistance, but I really don't have the time to make this happen. But I honestly believe that unless packaging has binary distribution support, it won't get "critical mass", on Windows if nowhere else.

Yeah, it's kinda required. We currently tell people to avoid pip because it doesn't support installation of binary distributions. It's just too hard to document the corner cases of using both pip and easy_install at the same time, or telling Windows people to use easy_install and UNIX people to use pip, or some combination thereof. I'm not sure if other features we (the Pylons project) make heavy use of like extras, console scripts, entry points, and testing hooks are currently supported, but if they aren't I'd be willing to help add them as necessary. We wouldn't really be able to use it without this stuff, in any case.

Just a thought - if it's where your interests lie, that would be great. Paul.

In a perfect world, I'd not be volunteering, but it looks like I have to if I want to be able to get to bitch. ;-) - C

On 05/15/2012 04:39 PM, Éric Araujo wrote:

Hi again,

...

On 1 May 2012 17:40, Chris McDonough wrote:

...

Is there a PEP for the "packaging" package? Is there any sort of unfinished business I can help with? AFAIK, there's no specific PEP for packaging (there are a number of related PEPs, but nothing specific like a roadmap). Yep. distutils2/packaging implement PEP 345 (Metadata 1.2), 376 (dist-info directory a.k.a. installation database) and 386 (version numbers), and also the older Metadata PEPs like distutils, but there was no PEP to discuss inclusion of d2 in Python 3.3: it just happened when a core developer (Martin von Löwis) indicated he was opposed to work on features (to add support for PEP 384 — Stable ABI for example) outside of the Python repository. I think that no PEP was asked by anyone because distutils2 is forked from code already in the stdlib and it implements accepted PEPs. There are small and big features added or in

Le 01/05/2012 14:28, Paul Moore a écrit : progress, many of them inspired by setuptools, that don’t have a PEP though.

As I said on my other reply there is no friendly list of issues or roadmap, only unsorted bugs and what’s in my mind.

OK. Is there a way for me to take a look at the unsorted bugs?

...

I'm sure Éric can give you much better pointers on what would be useful, but one issue I've tried to raise a few times, and more recently Jim Fulton raised here (http://mail.python.org/pipermail/distutils-sig/2012-March/018323.html) is that of binary distribution support in packaging2. I've never had the time to shepherd a proposal through beyond the "initial debate" stage, and I know it's not getting high on Éric's list of priorities, but it would be good to see some movement on this. Indeed, in private email with Paul I agreed on the importance of a binary distribution format and did a pre-publication review of his PEP, but we did not finish our discussion nor incorporated the alternate proposal that was discussed at PyCon and on the mailing list (which makes it hard to see a clear picture — PEPs are good :)

It seems unlikely that this hard topic can be solved for Python 3.3 / distutils2 1.0; what can be done however is to make sure that the extensibility hooks in d2 are well tested and documented so that when a bdist PEP reaches agreement and is implemented, a simple pysetup call and two lines of config will be all it takes to be able to use the new command.

I know that the situation is far from ideal, and far from our goals for 3.3, but anyway d2 was never intended as a full replacement for setuptools and pip (more on that in an upcoming reply to another of your messages when you listed the setuptools features used by Pyramid). I think packaging in Python 3.3 will be a first version put in the stdlib to gather feedback and reports, not a finished stable product.

I really don't want to add stop energy here, and I'm more than willing to row to get something going, but I'm afraid if that's the diagnosis, it means I'll personally have to oppose the inclusion of packaging in 3.3. We currently already have at least 3 competing solutions (setuptools, distribute, and distutils itself), and people are baffled about which to use and how to use it. Adding two more (packaging and distutils2) which are similarly semi-documented and which don't even solve the problems that the previous ones do would serve no purpose, and baking them into Python itself will mean they can't evolve in important ways. I'd suggest we just put the brakes on and slate something better for 3.4. Does that make sense, or does that make people sad? - C

On 5/16/12 5:01 AM, Alex Clark wrote:

Hi,

On 5/15/12 9:58 PM, Chris McDonough wrote:

...

On 05/15/2012 04:39 PM, Éric Araujo wrote:

...

Hi again,

...

On 1 May 2012 17:40, Chris McDonough wrote:

...

Is there a PEP for the "packaging" package? Is there any sort of unfinished business I can help with? AFAIK, there's no specific PEP for packaging (there are a number of related PEPs, but nothing specific like a roadmap). Yep. distutils2/packaging implement PEP 345 (Metadata 1.2), 376 (dist-info directory a.k.a. installation database) and 386 (version numbers), and also the older Metadata PEPs like distutils, but there was no PEP to discuss inclusion of d2 in Python 3.3: it just happened when a core developer (Martin von Löwis) indicated he was opposed to work on features (to add support for PEP 384 — Stable ABI for example) outside of the Python repository. I think that no PEP was asked by anyone because distutils2 is forked from code already in the stdlib and it implements accepted PEPs. There are small and big features added or in

Le 01/05/2012 14:28, Paul Moore a écrit : progress, many of them inspired by setuptools, that don’t have a PEP though.

As I said on my other reply there is no friendly list of issues or roadmap, only unsorted bugs and what’s in my mind.

OK. Is there a way for me to take a look at the unsorted bugs?

...

...

I'm sure Éric can give you much better pointers on what would be useful, but one issue I've tried to raise a few times, and more recently Jim Fulton raised here (http://mail.python.org/pipermail/distutils-sig/2012-March/018323.html)

is that of binary distribution support in packaging2. I've never had the time to shepherd a proposal through beyond the "initial debate" stage, and I know it's not getting high on Éric's list of priorities, but it would be good to see some movement on this. Indeed, in private email with Paul I agreed on the importance of a binary distribution format and did a pre-publication review of his PEP, but we did not finish our discussion nor incorporated the alternate proposal that was discussed at PyCon and on the mailing list (which makes it hard to see a clear picture — PEPs are good :)

It seems unlikely that this hard topic can be solved for Python 3.3 / distutils2 1.0; what can be done however is to make sure that the extensibility hooks in d2 are well tested and documented so that when a bdist PEP reaches agreement and is implemented, a simple pysetup call and two lines of config will be all it takes to be able to use the new command.

I know that the situation is far from ideal, and far from our goals for 3.3, but anyway d2 was never intended as a full replacement for setuptools and pip (more on that in an upcoming reply to another of your messages when you listed the setuptools features used by Pyramid). I think packaging in Python 3.3 will be a first version put in the stdlib to gather feedback and reports, not a finished stable product.

I really don't want to add stop energy here, and I'm more than willing to row to get something going, but I'm afraid if that's the diagnosis, it means I'll personally have to oppose the inclusion of packaging in 3.3. We currently already have at least 3 competing solutions (setuptools, distribute, and distutils itself), and people are baffled about which to use and how to use it. Adding two more (packaging and distutils2) which are similarly semi-documented and which don't even solve the problems that the previous ones do would serve no purpose, and baking them into Python itself will mean they can't evolve in important ways. I'd suggest we just put the brakes on and slate something better for 3.4. Does that make sense, or does that make people sad?

+1. I think that makes sense. I'd be willing to work on something if I felt like I understood what was going on, which I don't. For example, distribute's goal seems very clear: provide a well-maintained and frequently-released setuptools. It would make sense (to me) to include distribute in a future release of Python, to close the loop of the non-std-lib-packaging-lib-built-on-top-distutils-but-never-included-in-the-stdlib-situation.

But IIUC, it's too convoluted to attempt such a thing, and distutils2/packaging was meant to provide us the freedom to break backwards compat, but to what end? I don't think I fully understand the over-arching, real-world practical goals are for packaging in Python 3.x (other than to provide the pysetup command, and ini-style configuration over setup.py.)

This will never happen because Distribute's goal was 1/ to allow us to fix some bugs in setuptools and 2/ set some bridges for "packaging" adoption. But the final goal is to let it die where it is. packaging provides two things: - reference implementations for the PEPs we've written - that can be used by any tool - a replacement for distutils-the-tool, with improvements and backward compatible helpers, like a way to browse packages installed by other tools The only reason we did not do this in distutils was because any change I made in the code back then, even on private parts, was potentially breaking all the legacy code out there

Alex

...

- C _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

On 05/16/2012 02:35 AM, Tarek Ziadé wrote:

I know the idea of having packaging in the stdlib is something some people disliked, from day 1. And if I recall correctly, you did not like this either back then.

I've always been pro-stdlib-contains-a-package-installer-and-machinery, and I still am.

I am -1 on for two reasons:

1/ your argument about people being baffled about which tool to use will be worse if we work outside the stdlib. Having packaging in the stdlib and distutils frozen here, leads the path: "hey, the next tool in the stdlib is packaging"

It's fine to develop it inside the stdlib. It's harder to think about releasing it in 3.3 though without good docs. Do you think we have enough time to document it and define its scope and limitations and future plans in written form before 3.3beta1 ships?

2/ packaging is completely mature for some parts, like the PEP 386 implementation. And having the packaging.version module in the stdlib is a great step forward for standardization The pep 376 implementation is also providing great APIs to push all tools to inter-operate.

That's good to know.

If we develop this tool outside, I am afraid we will just add more confusion in the mix.

Can you see packaging as a set of utility at this point, rather than a tool that's going to replace instantly all the legacy tools ?

I can. - C

On 5/16/12 7:35 PM, Mark Sienkiewicz wrote:

...

Can you see packaging as a set of utility at this point, rather than a tool that's going to replace instantly all the legacy tools ?

If it is not going to replace the legacy tools, then why should I use it?

Who are you ? If you are a user that just installs software, you might want to use it because you can *uninstall* things. This is now less relevant now that pip has this feature, but easy_install does not. And with the PEP we wrote, any tool should be able to install/uninstall any project. And well, have this feature bundled into Python makes sense. If you are a packager for a project, you can describe in details your data files, and add more metadata that are understood by PyPI. If you are a debian packager, you will be able to define where the data files of a python project should be installed without having to patch some python code. I have much more but it's a bit hard to summarize this in a mail - I have a packaging fatigue anyways.

My understanding is that the legacy tools already work, for some vague definition of "work". Your task is to convince me that "packaging" is better -- which I think will be hard to do if packaging is not better.

It really depends what you put behind the word "better". And no, My task is not to convince you that packaging is better. My tasks were : 1 - try to define standards which every packaging tool can implement, for the sake of interoperability, have a consensus on them, have them "accepted" 2 - allow more "static" definitions of metadata 3 - add in the standard library a/ a reference implementation for all the standards b/ a minimal installer. 3-b is not ready for prime time obviously - but this should not take too long. Cheers Tarek

Mark S.

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

On Mon, May 21, 2012 at 3:42 PM, Tarek Ziadé wrote:

If you are a packager for a project, you can describe in details your data files, and add more metadata that are understood by PyPI.

If you are a debian packager, you will be able to define where the data files of a python project should be installed without having to patch some python code.

This right here is the killer feature for me. By my limited observations, most people solve the data files problem either by dumping their data files inside their python modules (which is an ugly abuse of the filesystem), or are simply using the cumbersome autotools in order to record the installation prefix for their data files. It blows my mind that it is standard practice for many GNOME apps written in Python to use a C compilation preprocessor in order to set a python variable so their python scripts can find their data files. Currently I am hacking distutils in order to accomplish this, like so: https://github.com/robru/gottengeography/blob/master/setup.py But this is ugly because it modifies the file in-place, so I always have to be careful not to accidentally commit the munged file into my git repo. It's absolutely critical that any replacement for distutils have built-in functionality for installed python code being able to query at run-time the location that data files were placed at install time. -- http://exolucere.ca

On Wed, May 23, 2012 at 1:48 AM, Tarek Ziadé wrote:

On 5/23/12 8:16 AM, Robert Park wrote:

...

It's absolutely critical that any replacement for distutils have built-in functionality for installed python code being able to query at run-time the location that data files were placed at install time.

Please read this section and let us know what you think:

http://docs.python.org/dev/packaging/setupcfg.html#resources

Yeah, the [files] definition in setup.cfg looks fine, the space syntax in the source definition I would consider highly important in order to avoid installing things into {datadir}/doc/foo/doc/ as given in the example.

This works in conjunction with the new sysconfig module, which can be configured system-wide by the linux distribution, or locally per projet

Then you can use an API to get the file from your code.

I didn't see any mention of this aspect in the documentation you linked. What does the API look like? -- http://exolucere.ca

I think the biggest thing missing wrt to command hooks is the ability to have the tool automatically generate a OS specific wrapper (.exe for windows, extension less with a shebang for *NIX). While I think it would be useful for this feature to live in std lib, I don't think it's near as required as getting the core of packaging into stdlib. Attempting to create packaging as outside of stdlib brings up images of http://xkcd.com/927/, and to me the only thing prevent packaging from just being another standard is the "Blessing" that comes with being in stdlib. On Wednesday, May 16, 2012 at 2:55 AM, Tarek Ziadé wrote:

On 5/16/12 3:58 AM, Chris McDonough wrote:

...

Adding two more (packaging and distutils2) which are similarly semi-documented and which don't even solve the problems that the previous ones do would serve no purpose, and baking them into Python itself will mean they can't evolve in important ways.

Oh, I think I need to answer to this too since you said you wanted to help. Packaging is not intended to be similar to setuptools in its features.

For instance we won't provide console scripts or entry points. The first one because 'script' is the same feature (except there's an indirection and I said before we could mimic this) The second one because we should build this kind of feature outside the stdlib (this is not something most people use, according to the survey I did back a few years ago, it's mostly zope/plone/repoze land)

I'd suggest you list what you can't do with "packaging" today and we work through that list to point which features are missing and should be developed *outside* the standard lib, and which ones are in "packaging" or should be

IOW: packaging should only be the common basis and provide a basic installer - not a full fledge tool you can use to replace the most advanced setuptools features. And we want it pluggable enough so people can build pluggable features on the top of it, like Eric explained earlier

Does that make sense ?

Cheers Tarek

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org (mailto:Distutils-SIG@python.org) http://mail.python.org/mailman/listinfo/distutils-sig

On 16 May 2012 07:55, Tarek Ziadé wrote:

...

I'd suggest you list what you can't do with "packaging" today and we work through that list to point which features are missing and should be developed *outside* the standard lib, and which ones are in "packaging" or should be This would be a very good step - but rather than simply getting responses in the mailing list, can I suggest that we need some sort of central location where the features still outstanding for packaging can be tracked. Call it a roadmap if you like. Maybe it should be a PEP - simply because I can't think of a better place to put it, but I'm open to suggestions (I don't think the bug tracker is the right place, fwiw). A wiki page sounds good, as long as each point turns into a bug at some

On 5/16/12 9:40 AM, Paul Moore wrote: point

At the moment, the biggest issue I see is that there are lots of discussions about what people believe is missing, but nothing clearly documenting what's intended to be there (and what is not - for example, your comment about entry points).

As a starter, my key "missing requirement" is support for binary distributions - whether this is a new "universal" format, or whether it is reusing the bdist_wininst/bdist_msi formats, I don't really care, but it needs to be formalised with a migration path, backward compatibility support considered, etc.

Do you feel like you could start the windows story section on that wiki page ?

...

IOW: packaging should only be the common basis and provide a basic installer - not a full fledge tool you can use to replace the most advanced setuptools features. And we want it pluggable enough so people can build pluggable features on the top of it, like Eric explained earlier

Does that make sense ? I would assume as a first guess that it should replace all of distutils, though? Ideally there should be no reason for people to use distutils for anything once packaging is available - am I right?

It does replace all distutils since it's a fork of it. The parts that is missing on purpose is bdist_rpm because we want anything RPM-related to be dealt outside the stdlib : there's a whole gap between how rhel/fedora guys do their packaging and what we offer with bdist_rpm, which seems to be perfect for... red hat 4 ? FWIW I have started the pypi2rpm project to group tools for RPM-izing python projects. Windows binaries were kept because it seems easy and feasible to maintain those from the stdlib -- but they still smell like old stuff because we don't have a Mr Windows in the distutils project. It was mostly Martin doing the legit work to make it work, and a couple of other people (forgive me if I forget someone) So... the windows binary story will improve if we get a champion for this. Cheers Tarek

Taking off Guido, I don't think he needs to be in the whole thread On 5/16/12 10:30 AM, Paul Moore wrote:

I'm happy to summarise this on there. Let me know a link to the wiki page. I created http://wiki.python.org/moin/Packaging/Roadmap

But 2 points:

1. Binary distributions is *not* a Windows issue. I originally pitched it as such, because that's where I care, but other people chimed in with an interest from a non-Windows POV. I can't comment on those requirements, though. And part of the problem is getting someone to make a unified statement. I drafted a PEP based around this, but Éric wanted to have a think about it, and it stalled. We also had other suggestions from Jim Fulton and Vinay Sanjip on various lists. So someone needs to take a lead on unifying all this. Yeah definitely agreed. Although it impacts windows the most because it's easier on other platform to deal with source archives.

2. The main Windows issue is that there are still a lot of bugs. Whenever I test stuff, I usually find myself hitting one or two bugs, often existing ones (I don't often get to report new bugs :-)) and then I'm stuck, unless I fix those bugs, which I often can't (or don't have the time to). So my efforts stall.

Yeah. And those bugs where in Distutils. What I mean is that Packaging has not improved much in this corner of distutils, but has not made it worse, and the tools are still accessible under the new version.

I'll have another try at using packaging in the 3.3a2 release and see what the picture looks like now. (Note - my key use case at the moment is pysetup install stuff from pypi, I'm not writing my own packages, so it's the backward compatibility with existing distutils-based packages that I'll be testing, and which is key for me). Ok thanks

...

Windows binaries were kept because it seems easy and feasible to maintain those from the stdlib -- but they still smell like old stuff because we don't have a Mr Windows in the distutils project.

It was mostly Martin doing the legit work to make it work, and a couple of other people (forgive me if I forget someone)

So... the windows binary story will improve if we get a champion for this. Agreed. Unfortunately, I don't have the time to be such a champion. But nevertheless, the current story with distutils/setuptools/distribute on Windows is perfectly fine, so unless packaging can get to that standard, I can't honestly suggest to people that they should switch over. It's a high bar to reach (particularly with such limited Windows expertise available) but necessary, IMO. Definitely ! I am afraid the scope is so wide that it's impossible to address all problems for v1

Another part were we lack a champion: compilers

Paul.

On 5/16/12 3:19 PM, Jim Fulton wrote:

On Wed, May 16, 2012 at 2:55 AM, Tarek Ziadé wrote:

...

On 5/16/12 3:58 AM, Chris McDonough wrote:

...

Adding two more (packaging and distutils2) which are similarly semi-documented and which don't even solve the problems that the previous ones do would serve no purpose, and baking them into Python itself will mean they can't evolve in important ways.

Oh, I think I need to answer to this too since you said you wanted to help. Packaging is not intended to be similar to setuptools in its features.

For instance we won't provide console scripts or entry points. The first one because 'script' is the same feature (except there's an indirection and I said before we could mimic this) I don't know what this means. Will we have something functionally-equivalent to console-scripts? Or will we have something more similar to the old distutils scripts functionality. If the later, then I doubt that packaging will work well with buildout.

Please explain us exactly what you mean with this feature. What we want to have in packaging is the old distutils srcipt feature, with an extra option: a way to point a callable instead of a module, and then a wrapper like setuptools does. If you need something else, please explain what it is. (there's no need to declare an entry point and iterate over all projects, just to find back a callable to run your script, right? you can just... import the callable)

Jim

On 05/16/2012 02:55 AM, Tarek Ziadé wrote:

On 5/16/12 3:58 AM, Chris McDonough wrote:

...

Adding two more (packaging and distutils2) which are similarly semi-documented and which don't even solve the problems that the previous ones do would serve no purpose, and baking them into Python itself will mean they can't evolve in important ways.

Oh, I think I need to answer to this too since you said you wanted to help. Packaging is not intended to be similar to setuptools in its features.

For instance we won't provide console scripts or entry points. The first one because 'script' is the same feature (except there's an indirection and I said before we could mimic this) The second one because we should build this kind of feature outside the stdlib (this is not something most people use, according to the survey I did back a few years ago, it's mostly zope/plone/repoze land)

I suspect many people don't really know they're using entry points when they are. PasteDeploy configuration files rely heavily on entry points. Those are used in various ways by Pylons, Pyramid, Zope, and Turbogears. (PasteDeploy itself been downloaded almost a million times from PyPI.) I'm fine with needing to depend on an external package to scan for entrypoint-like-things registered as the result of the installation of a distribution. But there will need to be a way to register entrypoint-like things (along the lines of arbitrary egg-info metadata) as the result of distribution installation using pysetup. Maybe that already exists.

I'd suggest you list what you can't do with "packaging" today and we work through that list to point which features are missing and should be developed *outside* the standard lib, and which ones are in "packaging" or should be

I can try.

IOW: packaging should only be the common basis and provide a basic installer - not a full fledge tool you can use to replace the most advanced setuptools features. And we want it pluggable enough so people can build pluggable features on the top of it, like Eric explained earlier

Does that make sense ?

I'd like to say it does, but I'm afraid it does not. I can see shipping both machinery and a full-fledged installer that handles all the use cases of existing installers. I can also see shipping just machinery and no installer at all. But I can't really see shipping both machinery and an installer that we know doesn't service use cases that existing installers already do. At least I can't see doing that in order to service an external shipping deadline. - C

On 05/21/2012 04:59 PM, Tarek Ziadé wrote:

On 5/16/12 6:26 PM, Chris McDonough wrote:

...

On 05/16/2012 02:55 AM, Tarek Ziadé wrote:

...

On 5/16/12 3:58 AM, Chris McDonough wrote:

...

Adding two more (packaging and distutils2) which are similarly semi-documented and which don't even solve the problems that the previous ones do would serve no purpose, and baking them into Python itself will mean they can't evolve in important ways.

Oh, I think I need to answer to this too since you said you wanted to help. Packaging is not intended to be similar to setuptools in its features.

For instance we won't provide console scripts or entry points. The first one because 'script' is the same feature (except there's an indirection and I said before we could mimic this) The second one because we should build this kind of feature outside the stdlib (this is not something most people use, according to the survey I did back a few years ago, it's mostly zope/plone/repoze land)

I suspect many people don't really know they're using entry points when they are. PasteDeploy configuration files rely heavily on entry points. Those are used in various ways by Pylons, Pyramid, Zope, and Turbogears. (PasteDeploy itself been downloaded almost a million times from PyPI.)

yes.

on a side note: The PyPI stats numbers are biased for many libraries because they are artificially increased by all the buildouts and Jenkins and Travis-CI calls out there. IOW: if a server downloads 100 times a day "PasteDeploy" because someone did not set a cache, does it make it 100 times more popular ? Same goes for any library that's a core part of the non-monolithic frameworks out there.

...

I'm fine with needing to depend on an external package to scan for entrypoint-like-things registered as the result of the installation of a distribution. But there will need to be a way to register entrypoint-like things (along the lines of arbitrary egg-info metadata) as the result of distribution installation using pysetup. Maybe that already exists.

we want to make the dist-info (see PEP 376) structure a directory where you can drop anything, so you can have arbitrary metadata. Right now, entry_points.txt is copied over in that directory by pysetup.

the new setup.cfg file allows you to define extra metadata as well, and what's awesome: we want to publish that static file to PyPI so tools can browse it !

=> http://docs.python.org/dev/packaging/setupcfg.html#extensibility

...

...

IOW: packaging should only be the common basis and provide a basic installer - not a full fledge tool you can use to replace the most advanced setuptools features. And we want it pluggable enough so people can build pluggable features on the top of it, like Eric explained earlier

Does that make sense ?

I'd like to say it does, but I'm afraid it does not.

I can see shipping both machinery and a full-fledged installer that handles all the use cases of existing installers. I can also see shipping just machinery and no installer at all.

But I can't really see shipping both machinery and an installer that we know doesn't service use cases that existing installers already do. At least I can't see doing that in order to service an external shipping deadline.

So what about this suggestion:

let's make this goal : pysetup should be able to install Pyramid (without [extras] sorry, we can live without it) - with a few adaptations in a branch if needed.

if we can make it work before 3.3rc1, great. If we fail we remove the installer part of packaging and move it to an external project

That's just a suggestion, I'll defer the decision to Eric because while I can help around, I don't have the time, neither the motivation to do packaging work these days.

But we have to hurry up - and check with Guido if he's ok with those decisions.

It sounds like a reasonable task for me to try out. In the meantime we've had a bit of a family emergency which will take some time to overcome and I won't be able to dedicate much energy to this. As a result, I have to declare bankruptcy here, so I'll have to live with whatever I get. I'm hoping though that someone else will step up here and do an evaluation and try to get things like Pyramid and popular Zope packages installed in a way that makes sense for straddling Python 2 and Python 3. I suspect if no one does this, it's going to be rough going. - C

For something like Pyramid the main piece that's missing is support for an entry_points like system.  As Tarek and other have pointed out the past, such a system could still be supported through a third-party plugin system that works via setup hooks and custom metadata.  I think there was even a prototype for something like that at one point...

'packaging' will be powerful enough to copy entry_points.txt into the .dist-info directory as a resource. Python bug #11880. The existing entry points code has a plugin mechanism for itself that will make it easy to iterate over entry_points.txt in dist-info as well as egg-info directories. Just write that plugin and you should be good to go. Though you might have to install the plugin in an egg-info directory at first...

On Thu, May 24, 2012 at 8:27 PM, PJ Eby wrote:

On Thu, May 24, 2012 at 6:56 PM, Daniel Holth wrote:

...

...

For something like Pyramid the main piece that's missing is support for an entry_points like system.  As Tarek and other have pointed out the past, such a system could still be supported through a third-party plugin system that works via setup hooks and custom metadata.  I think there was even a prototype for something like that at one point...

'packaging' will be powerful enough to copy entry_points.txt into the .dist-info directory as a resource. Python bug #11880.

The existing entry points code has a plugin mechanism for itself that will make it easy to iterate over entry_points.txt in dist-info as well as egg-info directories. Just write that plugin and you should be good to go. Though you might have to install the plugin in an egg-info directory at first...

What plugin mechanism are you talking about here, specifically?

For that matter, which "existing entry points code" are you talking about, too?  ;-)

Yeah, I'm a little confused by this too. Sure you could include an entry_points.txt in the dist-info. Though it's not clear to me that there's an entirely straightforward way to do this yet. I think some ideas have been tossed about for how to make it easier to include custom files in dist-info, but that there were no conclusions on that. Or maybe it already is easy and I'm just missing something. Then there would need to be some interface like pkg_resources.iter_entry_points() available to all software that relies on entry_points. This could be provided by the same plugin that adds entry_points.txt to dist-info. That plugin would have to be an installation requirement for any software that relies on it (as is setuptools for any package that currently uses entry_points, so I don't think that's such a hardship...) Erik

On Fri, May 25, 2012 at 2:24 PM, Daniel Holth wrote:

This discussion is confusing because we are either talking about a general plugin discovery mechanism for Python, or a plugin mechanism that packaging itself uses.

After the patch, entry_points.txt (which you are encouraged to write yourself) is copied by means of a new {dist-info} category using the resources = mechanism in packaging. This is a straightforward way to copy anything into the .dist-info directory. Apart from copying the file, 'packaging' has no role.

setup.cfg:

[files] packages = mypackage resources =    entry_points.txt={dist-info}

That's all there is to it.

Neat! I didn't realize there was a {dist-info} resource category. That, plus the register_finder() approach mentioned in the other thread could be a way to go for interoperability with packages using setuptools. It would still require a bit of hackery though... I suspect there would be resistance to adding support for this directly in packaging. But a setup_hook could set this up. Erik

On Sun, May 27, 2012 at 12:58 PM, PJ Eby wrote:

Is setup.cfg already in dist-info?  I guess if it isn't, you could just add it, using the same mechanism above.  Hm....  (goes to look at packaging docs)...  Ouch.  I'm seeing a bigger problem, which is that without either the ability to do "setup_requires" or to ship an sdist with a hook-altered setup.cfg, it doesn't look like you can actually implement all of setuptools' build functionality with just packaging.  (e.g. Setuptools can ship an sdist whose contents and version were determined using revision control info, but which does not then require the revision control tool when installing from the sdist, as the sdist contains a pre-built manifest, and a pre-built setup.cfg with hardcoded version numbers copied from the original revision control info.)

I've already got sort of a version of that for some of the packages I maintain here: https://svn.stsci.edu/trac/ssb/stsci_python/browser/stsci.distutils/trunk/st... It's a basic setup_hook that tacks the SVN revision on to version string in the metadata (and only if the version contains ".dev"). The SVN info itself comes from a generated module called 'version.py' that ships with the sdist and contains hard-coded version info and SVN revisions. The main reason it's a Python module is that it also contains an __version__ that can be imported by the main module of the package. This particular solution works for me. But the point is that it can be done pretty easily. However, the lack of a setup_requires-like feature still makes things pretty impossible short of shipping a copy of all the required setup hooks with the projects that use them. Certainly doable, but far from ideal.

I really hope I'm not going to have to write setuptools2 to work around these limitations.  ;-)   (Preferably, if I do write a setuptools2, I hope it can all be done using proper setup hooks on top of packaging/distutils2, without monkeypatching anything.)

Please, please no monkeypatching :)

On 05/29/2012 08:19 AM, PJ Eby wrote:

On Tue, May 29, 2012 at 10:20 AM, Erik Bray mailto:erik.m.bray@gmail.com> wrote:

This particular solution works for me. But the point is that it can be done pretty easily. However, the lack of a setup_requires-like feature still makes things pretty impossible short of shipping a copy of all the required setup hooks with the projects that use them. Certainly doable, but far from ideal.

Right, and I don't think distutils2 can really add setup_requires without blessing a package manager.

I'm confused by this statement. distutils2 _includes_ a package manager (pysetup); it has no need to bless an external one. What am I missing? Carl

On 05/29/2012 02:01 PM, PJ Eby wrote:

On Tue, May 29, 2012 at 1:24 PM, Carl Meyer wrote: On 05/29/2012 08:19 AM, PJ Eby wrote: > Right, and I don't think distutils2 can really add setup_requires > without blessing a package manager.

I'm confused by this statement. distutils2 _includes_ a package manager (pysetup); it has no need to bless an external one. What am I missing?

I might be confused; I haven't been following the goings-on of late with distutils2. At one point, I thought the plan was not to bless or include dependency-managing installers with the stdlib, or something like that. i.e., I thought the plan wasn't to support or bless full-service tools like buildout, easy_install, or pip, or anything comparable to them.

Right, yeah, the plans in this area were fluid for awhile, but the eventual conclusion was that the stdlib should have a command-line utility capable of installing packages with dependencies. That exists in default branch now; it's called pysetup. It doesn't have nearly all the features of easy_install, buildout, or pip, but it can install packages from an index with deps. Carl

On Tue, May 29, 2012 at 7:41 PM, PJ Eby wrote:

...

...

I might be confused; I haven't been following the goings-on of late with distutils2.  At one point, I thought the plan was not to bless or include dependency-managing installers with the stdlib, or something like that.  i.e., I thought the plan wasn't to support or bless full-service tools like buildout, easy_install, or pip, or anything comparable to them.

Right, yeah, the plans in this area were fluid for awhile, but the eventual conclusion was that the stdlib should have a command-line utility capable of installing packages with dependencies. That exists in default branch now; it's called pysetup. It doesn't have nearly all the features of easy_install, buildout, or pip, but it can install packages from an index with deps.

In any case, it still doesn't change the part where it's a good idea to ship a static setup.cfg, with hooks only needing to run on the sdist-building machine, unless they are actually part of the build process.  There are use cases for calculated data to be in the initial setup.cfg, where the calculation machinery doesn't need to be on the target (like generating the file list or version from revision control info).  So, a setup_requires (or maybe better named "build_requires") would still be helpful, but probably shouldn't be used for setup.cfg stability.

That's not a bad idea for certain kinds of metadata--version/vcs info for example. I like the idea of including a generated "static" setup.cfg in a source dist as a solution to that kind of problem. But that doesn't eliminate the need for setup_hooks (or even more complicated objects like custom commands) in an sdist. For example, the majority of projects I work on require Numpy to build one or two extension modules. They require hooks to check that the numpy package is importable, and then to use numpy's API to get the paths to the numpy headers and and them to the include_dirs for each extension module that requires them. That's not the only one though--one could have a whole suite of setup_hooks common to a bunch of projects. Custom Compiler classes are a possibility now too. One could ship a copy of those dependencies with each project, or have some kind of bootstrap script. But to be able to automatically download and add build dependencies to the path (a la setup_requires) would be much nicer. And packaging will have pysetup, so it should be doable. (Having the same capability for test dependencies and doc dependencies would be nice too, but not nearly as important). Erik

On Wed, May 30, 2012 at 12:19 PM, PJ Eby wrote:

On Wed, May 30, 2012 at 11:54 AM, Erik Bray wrote:

...

On Tue, May 29, 2012 at 7:41 PM, PJ Eby wrote:

...

...

...

I might be confused; I haven't been following the goings-on of late with distutils2.  At one point, I thought the plan was not to bless or include dependency-managing installers with the stdlib, or something like that.  i.e., I thought the plan wasn't to support or bless full-service tools like buildout, easy_install, or pip, or anything comparable to them.

Right, yeah, the plans in this area were fluid for awhile, but the eventual conclusion was that the stdlib should have a command-line utility capable of installing packages with dependencies. That exists in default branch now; it's called pysetup. It doesn't have nearly all the features of easy_install, buildout, or pip, but it can install packages from an index with deps.

In any case, it still doesn't change the part where it's a good idea to ship a static setup.cfg, with hooks only needing to run on the sdist-building machine, unless they are actually part of the build process.  There are use cases for calculated data to be in the initial setup.cfg, where the calculation machinery doesn't need to be on the target (like generating the file list or version from revision control info).  So, a setup_requires (or maybe better named "build_requires") would still be helpful, but probably shouldn't be used for setup.cfg stability.

That's not a bad idea for certain kinds of metadata--version/vcs info for example.  I like the idea of including a generated "static" setup.cfg in a source dist as a solution to that kind of problem.  But that doesn't eliminate the need for setup_hooks (or even more complicated objects like custom commands) in an sdist.

For example, the majority of projects I work on require Numpy to build one or two extension modules.  They require hooks to check that the numpy package is importable, and then to use numpy's API to get the paths to the numpy headers and and them to the include_dirs for each extension module that requires them.  That's not the only one though--one could have a whole suite of setup_hooks common to a bunch of projects.  Custom Compiler classes are a possibility now too.

One could ship a copy of those dependencies with each project, or have some kind of bootstrap script.  But to be able to automatically download and add build dependencies to the path (a la setup_requires) would be much nicer.  And packaging will have pysetup, so it should be doable.  (Having the same capability for test dependencies and doc dependencies would be nice too, but not nearly as important).

Certainly.  I was just saying that the generated-metadata cases need handling, too, and that people should also be informed that they don't need (and shouldn't use) setup_requires for simple metadata generation, and that it might be a good idea to call the feature "build_requires", and perhaps distinguish between "setup hooks" (for developers to have nice things) and "build hooks" (for stuff that absolutely has to run on the install machine).

I think we're on the same page then :) packaging/d2 also supports pre/post-command hooks which might, in most cases, be more appropriate for the "build hooks that absolutely have to run" case. In fact, I had completely forgotten this, but my aforementioned Numpy extension module hook is a pre-build_ext hook. That makes it absolutely clear that this is something we have to do before we can build an extension module, and that it doesn't have any purpose outside that context and shouldn't be executed ever time I run `pysetup something`. Erik

On 5/13/12 8:45 PM, Chris McDonough wrote:

On 05/01/2012 12:40 PM, Chris McDonough wrote:

...

Georg Brandl was trying to herd cats on outstanding PEPs today:

http://mail.python.org/pipermail/python-dev/2012-May/119157.html

Is there a PEP for the "packaging" package? Is there any sort of unfinished business I can help with?

Let's try this again. If distutils2/packaging is meant to actually be part of Python 3.3, I think it needs some documentation and implementation loving. Is there a list of outstanding issues that need resolution? I'm willing to row on them, but I need some direction.

I can't really see packaging being included in Python 3.3 without a good bit of work being put into it. Am I mistaken?

No you're not, we're quite late on what we need to finish -- I am unable to spend time on this these days, I defer directions to Eric. Maybe we could organize regular sprints to try to finish this

- C _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig