From chris.jerdonek at gmail.com Wed Oct 1 01:44:09 2014 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Tue, 30 Sep 2014 16:44:09 -0700 Subject: [Distutils] advice re: packaging tasks Message-ID: Hi, I was curious what others do for the following packaging tasks, or if you have any recommendations otherwise. There is also a code organization question at the end. 1) For starters, it's very easy to make mistakes in one's MANIFEST.in, so I hacked the sdist command in my setup.py to list the differences between one's project repo and the generated sdist each time you run "python setup.py sdist". Are there better solutions for this out there so I don't have to rely on my own hack? 2) Secondly, like many, my README files are in markdown, so I hacked a command in my setup.py to use Pandoc to convert README.md to a .rst file for use as the long_description argument to setup(). I also check in the resulting file for troubleshooting purposes, etc. Are there more elegant solutions for this that people know of? Also, for commands like the latter, is it better to define them in one's setup.py, or simply to have separate scripts in one's repo? Lastly, as these setup-related tasks grow larger and more complicated, I found it helped to break them out into a separate setup package that sits alongside my project's main package library (and even adding tests in some cases). Is this normal? Have other people run into this? Thanks, --Chris From donald at stufft.io Wed Oct 1 02:00:03 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 30 Sep 2014 20:00:03 -0400 Subject: [Distutils] advice re: packaging tasks In-Reply-To: References: Message-ID: > On Sep 30, 2014, at 7:44 PM, Chris Jerdonek wrote: > > Hi, > > I was curious what others do for the following packaging tasks, or if > you have any recommendations otherwise. There is also a code > organization question at the end. > > 1) For starters, it's very easy to make mistakes in one's MANIFEST.in, > so I hacked the sdist command in my setup.py to list the differences > between one's project repo and the generated sdist each time you run > "python setup.py sdist". Are there better solutions for this out > there so I don't have to rely on my own hack? https://warehouse.python.org/project/check-manifest/ > > 2) Secondly, like many, my README files are in markdown, so I hacked > a command in my setup.py to use Pandoc to convert README.md to a .rst > file for use as the long_description argument to setup(). I also > check in the resulting file for troubleshooting purposes, etc. Are > there more elegant solutions for this that people know of? Not that I?m aware of. > > Also, for commands like the latter, is it better to define them in > one's setup.py, or simply to have separate scripts in one's repo? I?d probably do it in the setup.py, or as an invoke task, see https://warehouse.python.org/project/invoke/. > > Lastly, as these setup-related tasks grow larger and more complicated, > I found it helped to break them out into a separate setup package that > sits alongside my project's main package library (and even adding > tests in some cases). Is this normal? Have other people run into > this? This is especially the case when I start using invoke. > > Thanks, > --Chris > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From olivier.grisel at ensta.org Wed Oct 1 09:03:15 2014 From: olivier.grisel at ensta.org (Olivier Grisel) Date: Wed, 1 Oct 2014 09:03:15 +0200 Subject: [Distutils] Microsoft Visual C++ Compiler for Python 2.7 In-Reply-To: <1607e5f43fc545f9bc48d35d576b6243@BN1PR0301MB0723.namprd03.prod.outlook.com> References: <416b70639b2c41a1a109fd14623e762b@DM2PR0301MB0734.namprd03.prod.outlook.com> <3d976f18b62d4db1a9b9da30976cdd85@DM2PR0301MB0734.namprd03.prod.outlook.com> <1607e5f43fc545f9bc48d35d576b6243@BN1PR0301MB0723.namprd03.prod.outlook.com> Message-ID: 2014-09-30 18:07 GMT+02:00 Steve Dower : > Paul Moore wrote: >> On 30 September 2014 16:56, Olivier Grisel wrote: >>> What is the story for project maintainers who want to also support >>> Python 3.3+ (for 32 bit and 64 bit python) for their project with >>> binary wheels for windows? >> >> It would be so easy at this point to ask "What's the chance of a similarly packaged >> version of VS2010 for Python 3.2/3.3/3.4?" But I really don't want Steve to get into >> any trouble with people saying "now look what you've started" :-) > > :-) > > The answer is basically no chance - the slippery slope was considered and shut down. > > If VC14 slips significantly and we have to stick with VC10 for Python 3.5, then I'll make the case again and see what we get, but for now the future story is to upgrade. Luckily, 3.3->3.5 is not going to be as hard as 2.7->3.5. Alright, sounds good. Thanks again. -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel From carl at oddbird.net Wed Oct 1 17:51:39 2014 From: carl at oddbird.net (Carl Meyer) Date: Wed, 01 Oct 2014 09:51:39 -0600 Subject: [Distutils] advice re: packaging tasks In-Reply-To: References: Message-ID: <542C230B.3050607@oddbird.net> Hi Chris, On 09/30/2014 05:44 PM, Chris Jerdonek wrote: > 2) Secondly, like many, my README files are in markdown, so I hacked > a command in my setup.py to use Pandoc to convert README.md to a .rst > file for use as the long_description argument to setup(). I also > check in the resulting file for troubleshooting purposes, etc. Are > there more elegant solutions for this that people know of? Perhaps this is stating the obvious, but the simplest solution here is just to switch your README to rst format. It's not that different from Markdown, GitHub/BitBucket will parse it and display it nicely, etc. Carl From david.genest at ubisoft.com Wed Oct 1 18:44:47 2014 From: david.genest at ubisoft.com (David Genest) Date: Wed, 1 Oct 2014 16:44:47 +0000 Subject: [Distutils] Wheels and dependent third party dlls on windows In-Reply-To: References: <469c55682b3d415e8a0bc272be536d2b@MSR-MAIL-EXCH01.ubisoft.org> <390b0a534a3042e79a74affb90f3cafe@MSR-MAIL-EXCH01.ubisoft.org> Message-ID: <7552171dcc124e57b0b446b7e381ae5e@MSR-MAIL-EXCH01.ubisoft.org> Thank you all for the precious info. Here are my observations: - We are merely writing extension modules with third party dependant code packaged in a dll. In my mind, this use case is not the exception, and would not necessarily warrant the use of a full blown solution like conda. Our deployed environments are self-contained. - If you run python setup.py bdist_wheel, the dlls specified in the scripts parameter end up in the wheel archive and does what is needed for our setup. (the dlls are copied to the scripts directory which is on PATH for the activated environment). - On the other hand, if you run pip wheel on the same package, the dlls are not placed in the archive. Is this a pip bug ? In an ideal world, the scripts directory would be called bin, like the unix counter-part, and any dependency, being startup scripts or dlls could be installed in the bin/ "environment global space". This path would be added to the python startup sequence (in order to not rely on the env's activate). I feel that the current state of affairs is not that far, because setup.py bdist_wheel works now. Knowing that there are alternatives on the way (in metadata 2.0 ?) and workarounds, we will go with our current wheel solution using setup.py bdist_wheel. If the bdist_wheel command ever loses the "package binary files in scripts dir" we have alternatives (listed in order of dependability): 1) add the dependent dlls to every package that needs it (Steve's answer https://mail.python.org/pipermail/distutils-sig/2014-September/024982.html concurs that the dependent dll would be loaded only once) 2) use technique to modify path in the first __init__.py (https://mail.python.org/pipermail/distutils-sig/2014-September/024962.html) 3) statically link the dependent library: not very good in the face of sharing code and having multiple copies in different extension modules with its state etc. Does the community still think this is a "I would not design my solution like yours" use-case ? The extension modules are a really good way to accelerate python, so they are bound to be constructed with other dependent libraries. It is not only a sdist world :-), particularly on Windows. Once again thanks, D. From Steve.Dower at microsoft.com Wed Oct 1 18:59:16 2014 From: Steve.Dower at microsoft.com (Steve Dower) Date: Wed, 1 Oct 2014 16:59:16 +0000 Subject: [Distutils] Wheels and dependent third party dlls on windows In-Reply-To: <7552171dcc124e57b0b446b7e381ae5e@MSR-MAIL-EXCH01.ubisoft.org> References: <469c55682b3d415e8a0bc272be536d2b@MSR-MAIL-EXCH01.ubisoft.org> <390b0a534a3042e79a74affb90f3cafe@MSR-MAIL-EXCH01.ubisoft.org> <7552171dcc124e57b0b446b7e381ae5e@MSR-MAIL-EXCH01.ubisoft.org> Message-ID: <59124bc81bc5400f830ca188a87e71be@BN1PR0301MB0723.namprd03.prod.outlook.com> David Genest wrote: > 1) add the dependent dlls to every package that needs it (Steve's answer > https://mail.python.org/pipermail/distutils-sig/2014-September/024982.html > concurs that the dependent dll would be loaded only once) This is the best approach regardless of what else works/doesn't work. Doing this will ensure that your extension module loads the correct dependencies regardless of what other packages or versions are installed in the same environment. The Scripts folder is not necessarily on the path for all users, and nor is the Python directory itself. Depending on how you launch Python, you may get DLLs loaded from the Python directory or the Scripts directory, but either way you should _always_ get DLLs loaded from the same directory as the extension module. Neither directory is intended to be a dumping ground for all the dependencies used by packages. I see no reason 'pip wheel' should produce different wheels to bdist_wheel (but perhaps there is one?), so I would consider this a bug. Cheers, Steve From p.f.moore at gmail.com Wed Oct 1 19:35:19 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 1 Oct 2014 18:35:19 +0100 Subject: [Distutils] Wheels and dependent third party dlls on windows In-Reply-To: <7552171dcc124e57b0b446b7e381ae5e@MSR-MAIL-EXCH01.ubisoft.org> References: <469c55682b3d415e8a0bc272be536d2b@MSR-MAIL-EXCH01.ubisoft.org> <390b0a534a3042e79a74affb90f3cafe@MSR-MAIL-EXCH01.ubisoft.org> <7552171dcc124e57b0b446b7e381ae5e@MSR-MAIL-EXCH01.ubisoft.org> Message-ID: On 1 October 2014 17:44, David Genest wrote: > - If you run python setup.py bdist_wheel, the dlls specified in the scripts parameter end up in the wheel archive and does what is needed for our setup. (the dlls are copied to the scripts directory which is on PATH for the activated environment). It sounds like you're using an old version of wheel. The --skip-scripts argument was removed (and skipping scripts made the default) in 0.23.0. > - On the other hand, if you run pip wheel on the same package, the dlls are not placed in the archive. Is this a pip bug ? No, this is not a pip bug. Scripts are omitted from wheels and generated on install from the metadata. DLLs aren't scripts, and putting them into the scripts list in setup.py will cause them to be treated inappropriately (as you see). > Knowing that there are alternatives on the way (in metadata 2.0 ?) and workarounds, we will go with our current wheel solution using setup.py bdist_wheel. You're likely to hit issues pretty soon, I suspect. You're using undefined (and generally strongly discouraged) behaviour, I'm afraid. But there *is* an intention to allow wheels to specify more possible locations for files to be installed (along the lines of the autoconf directory classes), so "the appropriate binary directory" should be a location you can specify in a supported manner in the longer term. > If the bdist_wheel command ever loses the "package binary files in scripts dir" we have alternatives (listed in order of dependability): > > 1) add the dependent dlls to every package that needs it (Steve's answer https://mail.python.org/pipermail/distutils-sig/2014-September/024982.html > concurs that the dependent dll would be loaded only once) > 2) use technique to modify path in the first __init__.py (https://mail.python.org/pipermail/distutils-sig/2014-September/024962.html) > 3) statically link the dependent library: not very good in the face of sharing code and having multiple copies in different extension modules with its state etc. > > Does the community still think this is a "I would not design my solution like yours" use-case ? The extension modules are a really good way to accelerate python, so they are bound to be constructed with other dependent libraries. It is not only a sdist world :-), particularly on Windows. I certainly wouldn't recommend using undefined behaviour like you are. Personally, I'd probably have designed my system around a single interface package that contained the DLL alongside a Python extension wrapping it. Other packages in your system could then depend on that one, and the DLLs would only be stored in one place. Other packages access the DLL via the extension (extensions can publish a C API for that purpose). But that's with hindsight, and learning the lessons from the issues you're having, so I wouldn't expect you to have known that in advance! Paul From cournape at gmail.com Wed Oct 1 19:53:31 2014 From: cournape at gmail.com (David Cournapeau) Date: Wed, 1 Oct 2014 18:53:31 +0100 Subject: [Distutils] Wheels and dependent third party dlls on windows In-Reply-To: <7552171dcc124e57b0b446b7e381ae5e@MSR-MAIL-EXCH01.ubisoft.org> References: <469c55682b3d415e8a0bc272be536d2b@MSR-MAIL-EXCH01.ubisoft.org> <390b0a534a3042e79a74affb90f3cafe@MSR-MAIL-EXCH01.ubisoft.org> <7552171dcc124e57b0b446b7e381ae5e@MSR-MAIL-EXCH01.ubisoft.org> Message-ID: On Wed, Oct 1, 2014 at 5:44 PM, David Genest wrote: > Thank you all for the precious info. > > Here are my observations: > > - We are merely writing extension modules with third party dependant code > packaged in a dll. In my mind, this use case is not the exception, and > would not necessarily warrant the use of a full blown solution like conda. > Our deployed environments are self-contained. > > - If you run python setup.py bdist_wheel, the dlls specified in the > scripts parameter end up in the wheel archive and does what is needed for > our setup. (the dlls are copied to the scripts directory which is on PATH > for the activated environment). > One issue with PATH is its global settings: if you copy your dll there and any other program the user may want to use, you will get into trouble. This is mostly an issue for well known 3rd party DLLs, though. > - On the other hand, if you run pip wheel on the same package, the dlls > are not placed in the archive. Is this a pip bug ? > > In an ideal world, the scripts directory would be called bin, like the > unix counter-part, and any dependency, being startup scripts or dlls could > be installed in the bin/ "environment global space". This path would be > added to the python startup sequence (in order to not rely on the env's > activate). > > I feel that the current state of affairs is not that far, because setup.py > bdist_wheel works now. > > Knowing that there are alternatives on the way (in metadata 2.0 ?) and > workarounds, we will go with our current wheel solution using setup.py > bdist_wheel. > > If the bdist_wheel command ever loses the "package binary files in scripts > dir" we have alternatives (listed in order of dependability): > > 1) add the dependent dlls to every package that needs it (Steve's answer > https://mail.python.org/pipermail/distutils-sig/2014-September/024982.html > concurs that the dependent dll would be loaded only once) > Note Steve's observation regarding manifests. It is not an uncommon issue when integrating with 3rd party libs in my experience (where renaming the dll is often impractical). David -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Wed Oct 1 21:02:40 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 1 Oct 2014 12:02:40 -0700 Subject: [Distutils] Wheels and dependent third party dlls on windows In-Reply-To: <7552171dcc124e57b0b446b7e381ae5e@MSR-MAIL-EXCH01.ubisoft.org> References: <469c55682b3d415e8a0bc272be536d2b@MSR-MAIL-EXCH01.ubisoft.org> <390b0a534a3042e79a74affb90f3cafe@MSR-MAIL-EXCH01.ubisoft.org> <7552171dcc124e57b0b446b7e381ae5e@MSR-MAIL-EXCH01.ubisoft.org> Message-ID: On Wed, Oct 1, 2014 at 9:44 AM, David Genest wrote: > - We are merely writing extension modules with third party dependent code > packaged in a dll. In my mind, this use case is not the exception, and > would not necessarily warrant the use of a full blown solution like conda. agreed -- it is not rare, so yes, it would be nice if the core python (pypa) systems addressed it. But like David said, Windows makes this really hard... > - If you run python setup.py bdist_wheel, the dlls specified in the > scripts parameter end up in the wheel archive and does what is needed for > our setup. (the dlls are copied to the scripts directory which is on PATH > for the activated environment). > If this is the PATH only for that environment, then this is probably fine. But one of the biggest sources of "dll hell" is that the same PATH is used for executables and dlls, and that dlls placed next to executables will be found. this means that any old app could find any old dll on teh PATH, and that there are a lot of dll on teh PATH. So putting dlls into the python "scripts" or "bin" dir is a bad idea in general -- who know what apps may find them? Couple this with the (absolutely incomprehensible to me) habit of folks to use short (still 8.3) names for dlls, without much version info, and you really have a mess. So if you do put your dlls into the Script dir -- do please give them nice long descriptive names! But isn't there a library or somethign directory where other python dlls are that could be used instead? then you could get clashes between python extensions, but it wouldn't clash with anything else on the system. > In an ideal world, the scripts directory would be called bin, like the > unix counter-part, why does the name matter at all? > and any dependency, being startup scripts or dlls could be installed in > the bin/ "environment global space". This path would be added to the python > startup sequence (in order to not rely on the env's activate). > ouch -- no dlls and top level scripts don't belong in the same place, period. ANother option is to make a python package that has little other than that dll in it, then yoru packaged list it as a dependency, and I _THINK_ there is some relative path magic that you can do so that your other extensions can find it. Anyone know what Anaconda does on Windows? > 1) add the dependent dlls to every package that needs it (Steve's answer > https://mail.python.org/pipermail/distutils-sig/2014-September/024982.html > concurs that the dependent dll would be loaded only once) > If Steve is correct, which he probably is -- this is a great way to go. Alternatively, alter my suggestion above a bit -- and have your "dll package" have a tiny extension that does nothing but link the dll in. then everything that depends on that dll will have a "import the_funny_dll_package" line at the top -- and this ends up looking just like a regular old python dependency. Again, make sure to use a descriptive enough name for the dll so that it doesn't clash with other packages (Not yours) that may use the same (similar) dll. Does the community still think this is a "I would not design my solution > like yours" use-case ? The extension modules are a really good way to > accelerate python, so they are bound to be constructed with other dependent > libraries. It is not only a sdist world :-), particularly on Windows. > this is common problem we'd all love to be able to solve! (and conda does help....) and sdist doesn't help anyway -- folks need to build and install it somehow anyway. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Wed Oct 1 22:06:26 2014 From: dholth at gmail.com (Daniel Holth) Date: Wed, 1 Oct 2014 16:06:26 -0400 Subject: [Distutils] Wheels and dependent third party dlls on windows In-Reply-To: References: <469c55682b3d415e8a0bc272be536d2b@MSR-MAIL-EXCH01.ubisoft.org> <390b0a534a3042e79a74affb90f3cafe@MSR-MAIL-EXCH01.ubisoft.org> <7552171dcc124e57b0b446b7e381ae5e@MSR-MAIL-EXCH01.ubisoft.org> Message-ID: On Wed, Oct 1, 2014 at 1:35 PM, Paul Moore wrote: > On 1 October 2014 17:44, David Genest wrote: >> - If you run python setup.py bdist_wheel, the dlls specified in the scripts parameter end up in the wheel archive and does what is needed for our setup. (the dlls are copied to the scripts directory which is on PATH for the activated environment). > > It sounds like you're using an old version of wheel. The > --skip-scripts argument was removed (and skipping scripts made the > default) in 0.23.0. > >> - On the other hand, if you run pip wheel on the same package, the dlls are not placed in the archive. Is this a pip bug ? > > No, this is not a pip bug. Scripts are omitted from wheels and > generated on install from the metadata. DLLs aren't scripts, and > putting them into the scripts list in setup.py will cause them to be > treated inappropriately (as you see). You are confusing generated entry_points script wrappers with the setup(scripts=...) scripts. The scripts=... scripts should never be skipped, even with --skip-scripts, they should work the same as they always have. From p.f.moore at gmail.com Wed Oct 1 22:12:15 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 1 Oct 2014 21:12:15 +0100 Subject: [Distutils] Wheels and dependent third party dlls on windows In-Reply-To: References: <469c55682b3d415e8a0bc272be536d2b@MSR-MAIL-EXCH01.ubisoft.org> <390b0a534a3042e79a74affb90f3cafe@MSR-MAIL-EXCH01.ubisoft.org> <7552171dcc124e57b0b446b7e381ae5e@MSR-MAIL-EXCH01.ubisoft.org> Message-ID: On 1 October 2014 21:06, Daniel Holth wrote: > You are confusing generated entry_points script wrappers with the > setup(scripts=...) scripts. The scripts=... scripts should never be > skipped, even with --skip-scripts, they should work the same as they > always have. Sorry, you're right. But the legacy (non entry-point) scripts are certainly fragile, and I'd recommend avoiding them. Even for actual scripts, and *certainly* as a hack to get things in the "Scripts" directory... Paul From ncoghlan at gmail.com Thu Oct 2 00:10:18 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 2 Oct 2014 08:10:18 +1000 Subject: [Distutils] Wheels and dependent third party dlls on windows In-Reply-To: References: <469c55682b3d415e8a0bc272be536d2b@MSR-MAIL-EXCH01.ubisoft.org> <390b0a534a3042e79a74affb90f3cafe@MSR-MAIL-EXCH01.ubisoft.org> <7552171dcc124e57b0b446b7e381ae5e@MSR-MAIL-EXCH01.ubisoft.org> Message-ID: On 2 Oct 2014 06:12, "Paul Moore" wrote: > > On 1 October 2014 21:06, Daniel Holth wrote: > > You are confusing generated entry_points script wrappers with the > > setup(scripts=...) scripts. The scripts=... scripts should never be > > skipped, even with --skip-scripts, they should work the same as they > > always have. > > Sorry, you're right. But the legacy (non entry-point) scripts are > certainly fragile, and I'd recommend avoiding them. Even for actual > scripts, and *certainly* as a hack to get things in the "Scripts" > directory... Note that PEP 459 currently proposes preserving this capability as "python.commands.prebuilt", so I personally consider it reasonable as a way of packaging arbitrary executables and non-entry-point based scripts. The main problem with using it for DLLs is the potential for "DLL hell" that you and others have mentioned, as version management on DLLs installed into shared directories can get very messy. Cheers, Nick. > > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Oct 2 01:16:46 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 2 Oct 2014 00:16:46 +0100 Subject: [Distutils] Wheels and dependent third party dlls on windows In-Reply-To: References: <469c55682b3d415e8a0bc272be536d2b@MSR-MAIL-EXCH01.ubisoft.org> <390b0a534a3042e79a74affb90f3cafe@MSR-MAIL-EXCH01.ubisoft.org> <7552171dcc124e57b0b446b7e381ae5e@MSR-MAIL-EXCH01.ubisoft.org> Message-ID: On 1 October 2014 23:10, Nick Coghlan wrote: >> Sorry, you're right. But the legacy (non entry-point) scripts are >> certainly fragile, and I'd recommend avoiding them. Even for actual >> scripts, and *certainly* as a hack to get things in the "Scripts" >> directory... > > Note that PEP 459 currently proposes preserving this capability as > "python.commands.prebuilt", so I personally consider it reasonable as a way > of packaging arbitrary executables and non-entry-point based scripts. Existing code tends to try to rewrite shebang lines on scripts. And maybe on Unix add the executable bit. The wheel spec (installing a wheel, the "Spread" step, item c) actually requires that script rewriting is done, so in theory that makes the scripts subdirectory in a wheel unsuitable. In practice of course, DLLs will be fine as they don't start with #!python. The "python.commands.prebuilt" spec doesn't say whether #! rewrites are done. If they are, they have the same issue as scripts. If they aren't, the current wheel spec doesn't offer an appropriate area to put them. Basically, it needs a wheel spec update to have an area which holds files that are copied unchanged to the destination Scripts folder. But this is entirely standards-style nitpicking. The existing scripts support is fine for DLLs, and will work as expected for every practical purpose. That's why I called it a "hack" - it works, but it's unofficial. Paul From david.genest at ubisoft.com Thu Oct 2 02:44:30 2014 From: david.genest at ubisoft.com (David Genest) Date: Thu, 2 Oct 2014 00:44:30 +0000 Subject: [Distutils] Wheels and dependent third party dlls on windows In-Reply-To: References: <469c55682b3d415e8a0bc272be536d2b@MSR-MAIL-EXCH01.ubisoft.org> <390b0a534a3042e79a74affb90f3cafe@MSR-MAIL-EXCH01.ubisoft.org> <7552171dcc124e57b0b446b7e381ae5e@MSR-MAIL-EXCH01.ubisoft.org> Message-ID: > Note that PEP 459 currently proposes preserving this capability as "python.commands.prebuilt", so I personally consider it reasonable as a way of packaging arbitrary >executables and non-entry-point based scripts. Yes, this will prove valuable (for other things than dlls, admittedly). > The main problem with using it for DLLs is the potential for "DLL hell" that you and others have mentioned, as version management on DLLs installed into shared directories > can get very messy. We control our environment and package only what is needed in it. This makes a micro system in which everything is controlled and isolated, even the global dlls (to the virtual env) I wanted to install. They become only accessible to the activated environment. I don?t see how it can become DLL hell in this situation. But hearing many fine and useful comments on the thread made me change my mind and package the dependency near the extension module. I still find that something like pep 459 will be really useful. To the list: Thanks for all you input. D. From marius at pov.lt Thu Oct 2 17:08:05 2014 From: marius at pov.lt (Marius Gedminas) Date: Thu, 2 Oct 2014 18:08:05 +0300 Subject: [Distutils] advice re: packaging tasks In-Reply-To: References: Message-ID: <20141002150805.GA28462@fridge.pov.lt> On Tue, Sep 30, 2014 at 04:44:09PM -0700, Chris Jerdonek wrote: > I was curious what others do for the following packaging tasks, or if > you have any recommendations otherwise. There is also a code > organization question at the end. > > 1) For starters, it's very easy to make mistakes in one's MANIFEST.in, > so I hacked the sdist command in my setup.py to list the differences > between one's project repo and the generated sdist each time you run > "python setup.py sdist". Are there better solutions for this out > there so I don't have to rely on my own hack? I wrote check-manifest for this. Prior to that I used to have horrific Makefile rules that did a similar check: https://github.com/mgedmin/objgraph/blob/master/Makefile#L69 > 2) Secondly, like many, my README files are in markdown, so I hacked > a command in my setup.py to use Pandoc to convert README.md to a .rst > file for use as the long_description argument to setup(). I also > check in the resulting file for troubleshooting purposes, etc. Are > there more elegant solutions for this that people know of? I wrote restview for a different purpose, but found it rather useful for discovering ReStructuredText problems that would make PyPI's fall back to plaintext rendering. If you run restview --long-description in a directory containing a setup.py, it'll invoke `python setup.py --long-description' and interpret it using the same docutils settings as PyPI, highlighting any errors. For a more automated (but perhaps less accurate) solution I pipe python setup.py --long-description | rst2html > /dev/null in my Makefile: https://github.com/mgedmin/objgraph/blob/master/Makefile#L99 > Also, for commands like the latter, is it better to define them in > one's setup.py, or simply to have separate scripts in one's repo? I'm really happy with being able to make releases of any of my packages with make release This runs the test suite (tox/detox FTW), checks for uncommitted changes, makes sure MANIFEST.in is correct, makes sure the version number in setup.py matches the version number in CHANGES.rst (and release date is today), makes sure there are no RST errors in long_descrption, and reminds me the commands I need to run to creates a version control system tag and package + upload the sdist to PyPI. (I'm too paranoid to let automated tools perform externally-visible actions like uploading to PyPI or pushing git tags, so I just copy & paste the commands once I'm sure everything's ship-shape.) There are other excellent tools for this (e.g. zest.releaser). One day I'll get tired of copy-pasting my Makefiles from project to project and I'll switch to zest.releaser. > Lastly, as these setup-related tasks grow larger and more complicated, > I found it helped to break them out into a separate setup package that > sits alongside my project's main package library (and even adding > tests in some cases). Is this normal? Have other people run into > this? I'm not sure what you mean. Do you have any examples? Marius Gedminas -- Life begins when you can spend your spare time programming instead of watching television. -- Cal Keegan -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From chris.jerdonek at gmail.com Thu Oct 2 18:15:54 2014 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Thu, 2 Oct 2014 09:15:54 -0700 Subject: [Distutils] advice re: packaging tasks In-Reply-To: <20141002150805.GA28462@fridge.pov.lt> References: <20141002150805.GA28462@fridge.pov.lt> Message-ID: On Thu, Oct 2, 2014 at 8:08 AM, Marius Gedminas wrote: > On Tue, Sep 30, 2014 at 04:44:09PM -0700, Chris Jerdonek wrote: >> I was curious what others do for the following packaging tasks, or if >> you have any recommendations otherwise. There is also a code >> organization question at the end. >> >> 1) For starters, it's very easy to make mistakes in one's MANIFEST.in, >> so I hacked the sdist command in my setup.py to list the differences >> between one's project repo and the generated sdist each time you run >> "python setup.py sdist". Are there better solutions for this out >> there so I don't have to rely on my own hack? > > I wrote check-manifest for this. Yes, I just started using this (as you know). :) It is very nice! > Prior to that I used to have horrific Makefile rules that did a similar check: > https://github.com/mgedmin/objgraph/blob/master/Makefile#L69 > >> 2) Secondly, like many, my README files are in markdown, so I hacked >> a command in my setup.py to use Pandoc to convert README.md to a .rst >> file for use as the long_description argument to setup(). I also >> check in the resulting file for troubleshooting purposes, etc. Are >> there more elegant solutions for this that people know of? > > I wrote restview for a different purpose, but found it rather useful for > discovering ReStructuredText problems that would make PyPI's fall back > to plaintext rendering. > > If you run > > restview --long-description > > in a directory containing a setup.py, it'll invoke `python setup.py > --long-description' and interpret it using the same docutils settings as > PyPI, highlighting any errors. > > For a more automated (but perhaps less accurate) solution I pipe > > python setup.py --long-description | rst2html > /dev/null Yes, I had been familiar with this latter approach. It is documented here: https://docs.python.org/2/distutils/packageindex.html#pypi-package-display It did not work for me in my case though. I eventually found that my use of URL fragments to link to sections elsewhere on the same page, and using relative links (supported by GitHub, https://github.com/blog/1395-relative-links-in-markup-files ) are what broke things. Here is a link to a PyPI bug report regarding making it easier to find such issues: https://bitbucket.org/pypa/pypi/issue/161/rest-formatting-fails-and-there-is-no-way If people are interested, I wrote a pandoc filter to convert such links to things that will continue to work once uploaded to PyPI: https://gist.github.com/cjerdonek/76608610df43fd5b0fc3 > in my Makefile: > https://github.com/mgedmin/objgraph/blob/master/Makefile#L99 > >> Also, for commands like the latter, is it better to define them in >> one's setup.py, or simply to have separate scripts in one's repo? > > I'm really happy with being able to make releases of any of my packages > with > > make release > > This runs the test suite (tox/detox FTW), checks for uncommitted > changes, makes sure MANIFEST.in is correct, makes sure the version > number in setup.py matches the version number in CHANGES.rst (and > release date is today), makes sure there are no RST errors in > long_descrption, and reminds me the commands I need to run to creates a > version control system tag and package + upload the sdist to PyPI. (I'm > too paranoid to let automated tools perform externally-visible actions > like uploading to PyPI or pushing git tags, so I just copy & paste the > commands once I'm sure everything's ship-shape.) That sounds great. > > There are other excellent tools for this (e.g. zest.releaser). One day > I'll get tired of copy-pasting my Makefiles from project to project and > I'll switch to zest.releaser. > >> Lastly, as these setup-related tasks grow larger and more complicated, >> I found it helped to break them out into a separate setup package that >> sits alongside my project's main package library (and even adding >> tests in some cases). Is this normal? Have other people run into >> this? > > I'm not sure what you mean. Do you have any examples? I mean that if you are working on project MyProject with package myproject inside the repo, you might find yourself adding ad hoc custom code to setup.py. If this setup.py code starts to grow (a bit like your Makefile may grow), it might make sense to move some of the setup code to a package called something like myproject_setup alongside myproject (which would be used only for setup tasks). And if this setup code was sufficiently complicated, you might find yourself wanting to add unit tests for some of it, so you might have myproject_setup/tests (and even testing it as part of your automated tests, etc). --Chris > > Marius Gedminas > -- > Life begins when you can spend your spare time programming instead of > watching television. > -- Cal Keegan > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > From chris.barker at noaa.gov Thu Oct 2 18:12:13 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 2 Oct 2014 09:12:13 -0700 Subject: [Distutils] Wheels and dependent third party dlls on windows In-Reply-To: References: <469c55682b3d415e8a0bc272be536d2b@MSR-MAIL-EXCH01.ubisoft.org> <390b0a534a3042e79a74affb90f3cafe@MSR-MAIL-EXCH01.ubisoft.org> <7552171dcc124e57b0b446b7e381ae5e@MSR-MAIL-EXCH01.ubisoft.org> Message-ID: On Wed, Oct 1, 2014 at 5:44 PM, David Genest wrote: > We control our environment and package only what is needed in it. This > makes a micro system in which everything is controlled and isolated, even > the global dlls (to the virtual env) I wanted to install. If that is your use case, you may want to take a good lok at conda -- that is exactly what it is for -- why re-invent the wheel? ( sorry ). Note that while conda is the package manger for Anaconda, it can also be used to build your own distribution, you wouldn't need to adopt Anaconda as a platform. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Fri Oct 3 08:05:50 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 3 Oct 2014 02:05:50 -0400 Subject: [Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting Message-ID: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> Here?s round 4 of PEP 470, I believe I?ve addressed the comments from the previous thread. I've also tried to clarify the text as well as the motivations better. You can view this online at: https://www.python.org/dev/peps/pep-0470/ --------------------- PEP: 470 Title: Using Multi Repository Support for External to PyPI Package File Hosting Version: $Revision$ Last-Modified: $Date$ Author: Donald Stufft , BDFL-Delegate: Richard Jones Discussions-To: distutils-sig at python.org Status: Draft Type: Process Content-Type: text/x-rst Created: 12-May-2014 Post-History: 14-May-2014, 05-Jun-2014, 03-Oct-2014 Replaces: 438 Abstract ======== This PEP proposes a mechanism for project authors to register with PyPI an external repository where their project's downloads can be located. This information can than be included as part of the simple API so that installers can use it to tell users where the item they are attempting to install is located and what they need to do to enable this additional repository. In addition to adding discovery information to make explicit multiple repositories easy to use, this PEP also deprecates and removes the implicit multiple repository support which currently functions through directly or indirectly linking offsite via the simple API. Finally this PEP also proposes deprecating and removing the functionality added by PEP 438, particularly the additional rel information and the meta tag to indicate the API version. This PEP *does* not propose mandating that all authors upload their projects to PyPI in order to exist in the index nor does it propose any change to the human facing elements of PyPI. Rationale ========= Historically PyPI did not have any method of hosting files nor any method of automatically retrieving installables, it was instead focused on providing a central registry of names, to prevent naming collisions, and as a means of discovery for finding projects to use. In the course of time setuptools began to scrape these human facing pages, as well as pages linked from those pages, looking for things it could automatically download and install. Eventually this became the "Simple" API which used a similar URL structure however it eliminated any of the extraneous links and information to make the API more efficient. Additionally PyPI grew the ability for a project to upload release files directly to PyPI enabling PyPI to act as a repository in addition to an index. This gives PyPI two equally important roles that it plays in the Python ecosystem, that of index to enable easy discovery of Python projects and central repository to enable easy hosting, download, and installation of Python projects. Due to the history behind PyPI and the very organic growth it has experienced the lines between these two roles are blurry, and this blurriness has caused confusion for the end users of both of these roles and this has in turn caused ire between people attempting to use PyPI in different capacities, most often when end users want to use PyPI as a repository but the author wants to use PyPI soley as an index. By moving to using explict multiple repositories we can make the lines between these two roles much more explicit and remove the "hidden" surprises caused by the current implementation of handling people who do not want to use PyPI as a repository. However simply moving to explicit multiple repositories is a regression in discoverablity, and for that reason this PEP adds an extension to the current simple API which will enable easy discovery of the specific repository that a project can be found in. PEP 438 attempted to solve this issue by allowing projects to explicitly declare if they were using the repository features or not, and if they were not, it had the installers classify the links it found as either "internal", "verifiable external" or "unverifiable external". PEP 438 was accepted and implemented in pip 1.4 (released on Jul 23, 2013) with the final transition implemented in pip 1.5 (released on Jan 2, 2014). PEP 438 was successful in bringing about more people to utilize PyPI's repository features, an altogether good thing given the global CDN powering PyPI providing speed ups for a lot of people, however it did so by introducing a new point of confusion and pain for both the end users and the authors. Why Additional Repositories? ---------------------------- The two common installer tools, pip and easy_install/setuptools, both support the concept of additional locations to search for files to satisify the installation requirements and have done so for many years. This means that there is no need to "phase" in a new flag or concept and the solution to installing a project from a repository other than PyPI will function regardless of how old (within reason) the end user's installer is. Not only has this concept existed in the Python tooling for some time, but it is a concept that exists across languages and even extending to the OS level with OS package tools almost universally using multiple repository support making it extremely likely that someone is already familar with the concept. Additionally, the multiple repository approach is a concept that is useful outside of the narrow scope of allowing projects which wish to be included on the index portion of PyPI but do not wish to utilize the repository portion of PyPI. This includes places where a company may wish to host a repository that contains their internal packages or where a project may wish to have multiple "channels" of releases, such as alpha, beta, release candidate, and final release. Setting up an external repository is very simple, it can be achieved with nothing more than a filesystem, some files to host, and any web server capable of serving files and generating an automated index of directories (commonly called "autoindex"). This can be as simple as: :: $ mkdir -p /var/www/index.example.com/ $ mkdir -p /var/www/index.example.com/myproject/ $ mv ~/myproject-1.0.tar.gz /var/www/index.example.com/myproject/ $ twistd -n web --path /var/www/index.example.com/ Using this additional location within pip is also simple and can be included on a per invocation, per shell, or per user basis. The pip 6.0 will also include the ability to configure this on a per virtual environment or per machine basis as well. This can be as simple as: :: $ # As a CLI argument $ pip install --extra-index-url https://index.example.com/ myproject $ # As an environment variable $ PIP_EXTRA_INDEX_URL=https://pypi.example.com/ pip install myproject $ # With a configuration file $ echo "[global]\nextra-index-url = https://pypi.example.com/" > ~/.pip/pip.conf $ pip install myproject Why Not PEP 438 or Similar? --------------------------- While the additional search location support has existed in pip and setuptools for quite some time support for PEP 438 has only existed in pip since the 1.4 version, and still has yet to be implemented in setuptools. The design of PEP 438 did mean that users still benefited for projects which did not require external files even with older installers, however for projects which *did* require external files, users are still silently being given either potentionally unreliable or, even worse, unsafe files to download. This system is also unique to Python as it arises out of the history of PyPI, this means that it is almost certain that this concept will be foreign to most, if not all users, until they encounter it while attempting to use the Python toolchain. Additionally, the classification system proposed by PEP 438 has, in practice, turned out to be extremely confusing to end users, so much so that it is a position of this PEP that the situation as it stands is completely untenable. The common pattern for a user with this system is to attempt to install a project possibly get an error message (or maybe not if the project ever uploaded something to PyPI but later switched without removing old files), see that the error message suggests ``--allow-external``, they reissue the command adding that flag most likely getting another error message, see that this time the error message suggests also adding ``--allow-unverified``, and again issue the command a third time, this time finally getting the thing they wish to install. This UX failure exists for several reasons. 1. If pip can locate files at all for a project on the Simple API it will simply use that instead of attempting to locate more. This is generally the right thing to do as attempting to locate more would erase a large part of the benefit of PEP 438. This means that if a project *ever* uploaded a file that matches what the user has requested for install that will be used regardless of how old it is. 2. PEP 438 makes an implicit assumption that most projects would either upload themselves to PyPI or would update themselves to directly linking to release files. While a large number of projects *did* ultimately decide to upload to PyPI, some of them did so only because the UX around what PEP 438 was so bad that they felt forced to do so. More concerning however, is the fact that very few projects have opted to directly and safely link to files and instead they still simply link to pages which must be scraped in order to find the actual files, thus rendering the safe variant (``--allow-external``) largely useless. 3. Even if an author wishes to directly link to their files, doing so safely is non-obvious. It requires the inclusion of a MD5 hash (for historical reasons) in the hash of the URL. If they do not include this then their files will be considered "unverified". 4. PEP 438 takes a security centric view and disallows any form of a global opt in for unverified projects. While this is generally a good thing, it creates extremely verbose and repetive command invocations such as: :: $ pip install --allow-external myproject --allow-unverified myproject myproject $ pip install --allow-all-external --allow-unverified myproject myproject Multiple Repository/Index Support ================================= Installers SHOULD implement or continue to offer, the ability to point the installer at multiple URL locations. The exact mechanisms for a user to indicate they wish to use an additional location is left up to each indidivdual implementation. Additionally the mechanism discovering an installation candidate when multiple repositories are being used is also up to each individual implementation, however once configured an implementation should not discourage, warn, or otherwise cast a negative light upon the use of a repository simply because it is not the default repository. Currently both pip and setuptools implement multiple repository support by using the best installation candidate it can find from either repository, essentially treating it as if it were one large repository. Installers SHOULD also implement some mechanism for removing or otherwise disabling use of the default repository. The exact specifics of how that is achieved is up to each indidivdual implementation. End users wishing to limit what files they pull from which repository can simply use `devpi `_ to whitelist projects from PyPI or another repository. External Index Discovery ======================== One of the problems with using an additional index is one of discovery. Users will not generally be aware that an additional index is required at all much less where that index can be found. Projects can attempt to convey this information using their description on the PyPI page however that excludes people who discover their project organically through ``pip search``. To support projects that wish to externally host their files and to enable users to easily discover what additional indexes are required, PyPI will gain the ability for projects to register external index URLs along with an associated comment for each. These URLs will be made available on the simple page however they will not be linked or provided in a form that older installers will automatically search them. This ability will take the form of a ```` tag. The name of this tag must be set to ``external-repository`` and the content will be a link to the location of the external repository. An optional data-description attribute will convey any comments or description that the author has provided. An example would look something like: :: When an external repository is added to a project, new uploads will no longer be permitted to that project. However any existing files will simply be hidden from the simple API and the web interface until all of the external repositories are removed, in which case they will be visible again. PyPI MUST warn authors if adding an external repository will hide files and that warning must persist on any of the project management pages for that particular project. When an installer fetches the simple page for a project, if it finds this additional meta-data and it cannot find any files for that project in it's configured URLs then it should use this data to tell the user how to add one or more of the additional URLs to search in. This message should include any comments that the project has included to enable them to communicate to the user and provide hints as to which URL they might want (e.g. if some are only useful or compatible with certain platforms or situations). When the installer has implemented the auto discovery mechanisms they should also deprecate any of the mechanisms added for PEP 438 (such as ``--allow-external``) for removal at the end of the deprecation period proposed by the PEP. This feature *must* be added to PyPI prior to starting the deprecation and removal process for the implicit offsite hosting functionality. Deprecation and Removal of Link Spidering ========================================= A new hosting mode will be added to PyPI. This hosting mode will be called ``pypi-only`` and will be in addition to the three that PEP 438 has already given us which are ``pypi-explicit``, ``pypi-scrape``, ``pypi-scrape-crawl``. This new hosting mode will modify a project's simple api page so that it only lists the files which are directly hosted on PyPI and will not link to anything else. Upon acceptance of this PEP and the addition of the ``pypi-only`` mode, all new projects will be defaulted to the PyPI only mode and they will be locked to this mode and unable to change this particular setting. ``pypi-only`` projects will still be able to register external index URLs as described above - the "pypi-only" refers only to the download links that are published directly on PyPI. An email will then be sent out to all of the projects which are hosted only on PyPI informing them that in one month their project will be automatically converted to the ``pypi-only`` mode. A month after these emails have been sent any of those projects which were emailed, which still are hosted only on PyPI will have their mode set to ``pypi-only``. After that switch, an email will be sent to projects which rely on hosting external to PyPI. This email will warn these projects that externally hosted files have been deprecated on PyPI and that in 6 months from the time of that email that all external links will be removed from the installer APIs. This email *must* include instructions for converting their projects to be hosted on PyPI and *must* include links to a script or package that will enable them to enter their PyPI credentials and package name and have it automatically download and re-host all of their files on PyPI. This email *must also* include instructions for setting up their own index page and registering that with PyPI, including the fact that they can use pythonhosted.org as a host for an index page without requiring them to host any additional infrastructure or purchase a TLS certificate. This email must also contain a link to the Terms of Service for PyPI as many users may have signed up a long time ago and may not recall what those terms are. Five months after the initial email, another email must be sent to any projects still relying on external hosting. This email will include all of the same information that the first email contained, except that the removal date will be one month away instead of six. Finally a month later all projects will be switched to the ``pypi-only`` mode and PyPI will be modified to remove the externally linked files functionality. At this point in time any installers should finally remove any of the deprecated PEP 438 functionality such as ``--allow-external`` and ``--allow-unverified`` in pip. Impact ====== The largest impact of this is going to be projects where the maintainers are no longer maintaining the project, for one reason or another. For these projects it's unlikely that a maintainer will arrive to set the external index metadata which would allow the auto discovery mechanism to find it. Looking at the numbers factoring out PIL (which has been special cased below) the actual impact should be quite low, with it affecting just 3.8% of projects which host any files only externally or 2.2% which have their latest version hosted only externally. 6674 unique IP addresses have accessed the Simple API for these 3.8% of projects in a single day (2014-09-30). Of those, 99.5% of them installed something which could not be verified, and thus they were open to a Remote Code Execution via a Man-In-The-Middle attack, while 7.9% installed something which could be verified and only 0.4% only installed things which could be verified. Projects Which Rely on Externally Hosted files ---------------------------------------------- This is determined by crawling the simple index and looking for installable files using a similar detection method as pip and setuptools use. The "latest" version is determined using ``pkg_resources.parse_version`` sort order and it is used to show whether or not the latest version is hosted externally or only old versions are. ============ ======= ================ =================== ======= \ PyPI External (old) External (latest) Total ============ ======= ================ =================== ======= **Safe** 43313 16 39 43368 **Unsafe** 0 756 1092 1848 **Total** 43313 772 1131 45216 ============ ======= ================ =================== ======= Top Externally Hosted Projects by Requests ------------------------------------------ This is determined by looking at the number of requests the ``/simple//`` page had gotten in a single day. The total number of requests during that day was 10,623,831. ============================== ======== Project Requests ============================== ======== PIL 63869 Pygame 2681 mysql-connector-python 1562 pyodbc 724 elementtree 635 salesforce-python-toolkit 316 wxPython 295 PyXML 251 RBTools 235 python-graph-core 123 cElementTree 121 ============================== ======== Top Externally Hosted Projects by Unique IPs -------------------------------------------- This is determined by looking at the IP addresses of requests the ``/simple//`` page had gotten in a single day. The total number of unique IP addresses during that day was 124,604. ============================== ========== Project Unique IPs ============================== ========== PIL 4553 mysql-connector-python 462 Pygame 202 pyodbc 181 elementtree 166 wxPython 126 RBTools 114 PyXML 87 salesforce-python-toolkit 76 pyDes 76 ============================== ========== PIL --- It's obvious from the numbers above that the vast bulk of the impact come from the PIL project. On 2014-05-17 an email was sent to the contact for PIL inquiring whether or not they would be willing to upload to PyPI. A response has not been received as of yet (2014-10-03) nor has any change in the hosting happened. Due to the popularity of PIL this PEP also proposes that during the deprecation period that PyPI Administrators will set the PIL download URL as the external index for that project. Allowing the users of PIL to take advantage of the auto discovery mechanisms although the project has seemingly become unmaintained. Rejected Proposals ================== Keep the current classification system but adjust the options ------------------------------------------------------------- This PEP rejects several related proposals which attempt to fix some of the usability problems with the current system but while still keeping the general gist of PEP 438. This includes: * Default to allowing safely externally hosted files, but disallow unsafely hosted. * Default to disallowing safely externally hosted files with only a global flag to enable them, but disallow unsafely hosted. * Continue on the suggested path of PEP 438 and remove the option to unsafely host externally but continue to allow the option to safely host externally. These proposals are rejected because: * The classification system introduced in PEP 438 in an entirely unique concept to PyPI which is not generically applicable even in the context of Python packaging. Adding additional concepts comes at a cost. * The classification system itself is non-obvious to explain and to pre-determine what classification of link a project will require entails inspecting the project's ``/simple//`` page, and possibly any URLs linked from that page. * The ability to host externally while still being linked for automatic discovery is mostly a historic relic which causes a fair amount of pain and complexity for little reward. * The installer's ability to optimize or clean up the user interface is limited due to the nature of the implicit link scraping which would need to be done. This extends to the ``--allow-*`` options as well as the inability to determine if a link is expected to fail or not. * The mechanism paints a very broad brush when enabling an option, while PEP 438 attempts to limit this with per package options. However a project that has existed for an extended period of time may often times have several different URLs listed in their simple index. It is not unsusual for at least one of these to no longer be under control of the project. While an unregistered domain will sit there relatively harmless most of the time, pip will continue to attempt to install from it on every discovery phase. This means that an attacker simply needs to look at projects which rely on unsafe external URLs and register expired domains to attack users. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From holger at merlinux.eu Fri Oct 3 10:06:29 2014 From: holger at merlinux.eu (holger krekel) Date: Fri, 3 Oct 2014 08:06:29 +0000 Subject: [Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting In-Reply-To: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> References: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> Message-ID: <20141003080629.GG7954@merlinux.eu> Hi Donald, i could just only briefly glimpse over the new draft. I am still not in favor of the PEP because it forces backard-incompatible changes and work on various sides for not enough gain. Particularly end users will see previously working commands now fail and if they run a new enough pip version they get a hint on how to manually fix it. In the longer term you argue that people will appreciate reusing the concept of dealing with multiple repositories as known with Linux Distros. And that PEP470 would eventually simplify the user interface of pip and the implementation of pypi a bit. I'll see next week if i can come up with suggestions how to reduce friction introduced by the PEP while maintaining the benefits. And probably with a few questions because i didn't understand all details yet i think. best, holger On Fri, Oct 03, 2014 at 02:05 -0400, Donald Stufft wrote: > Here?s round 4 of PEP 470, I believe I?ve addressed the comments from the > previous thread. I've also tried to clarify the text as well as the motivations > better. > > You can view this online at: https://www.python.org/dev/peps/pep-0470/ > > --------------------- > > PEP: 470 > Title: Using Multi Repository Support for External to PyPI Package File Hosting > Version: $Revision$ > Last-Modified: $Date$ > Author: Donald Stufft , > BDFL-Delegate: Richard Jones > Discussions-To: distutils-sig at python.org > Status: Draft > Type: Process > Content-Type: text/x-rst > Created: 12-May-2014 > Post-History: 14-May-2014, 05-Jun-2014, 03-Oct-2014 > Replaces: 438 > > > Abstract > ======== > > This PEP proposes a mechanism for project authors to register with PyPI an > external repository where their project's downloads can be located. This > information can than be included as part of the simple API so that installers > can use it to tell users where the item they are attempting to install is > located and what they need to do to enable this additional repository. In > addition to adding discovery information to make explicit multiple repositories > easy to use, this PEP also deprecates and removes the implicit multiple > repository support which currently functions through directly or indirectly > linking offsite via the simple API. Finally this PEP also proposes deprecating > and removing the functionality added by PEP 438, particularly the additional > rel information and the meta tag to indicate the API version. > > This PEP *does* not propose mandating that all authors upload their projects to > PyPI in order to exist in the index nor does it propose any change to the human > facing elements of PyPI. > > > Rationale > ========= > > Historically PyPI did not have any method of hosting files nor any method of > automatically retrieving installables, it was instead focused on providing a > central registry of names, to prevent naming collisions, and as a means of > discovery for finding projects to use. In the course of time setuptools began > to scrape these human facing pages, as well as pages linked from those pages, > looking for things it could automatically download and install. Eventually this > became the "Simple" API which used a similar URL structure however it > eliminated any of the extraneous links and information to make the API more > efficient. Additionally PyPI grew the ability for a project to upload release > files directly to PyPI enabling PyPI to act as a repository in addition to an > index. > > This gives PyPI two equally important roles that it plays in the Python > ecosystem, that of index to enable easy discovery of Python projects and > central repository to enable easy hosting, download, and installation of Python > projects. Due to the history behind PyPI and the very organic growth it has > experienced the lines between these two roles are blurry, and this blurriness > has caused confusion for the end users of both of these roles and this has in > turn caused ire between people attempting to use PyPI in different capacities, > most often when end users want to use PyPI as a repository but the author wants > to use PyPI soley as an index. > > By moving to using explict multiple repositories we can make the lines between > these two roles much more explicit and remove the "hidden" surprises caused > by the current implementation of handling people who do not want to use PyPI > as a repository. However simply moving to explicit multiple repositories is > a regression in discoverablity, and for that reason this PEP adds an extension > to the current simple API which will enable easy discovery of the specific > repository that a project can be found in. > > PEP 438 attempted to solve this issue by allowing projects to explicitly > declare if they were using the repository features or not, and if they were > not, it had the installers classify the links it found as either "internal", > "verifiable external" or "unverifiable external". PEP 438 was accepted and > implemented in pip 1.4 (released on Jul 23, 2013) with the final transition > implemented in pip 1.5 (released on Jan 2, 2014). > > PEP 438 was successful in bringing about more people to utilize PyPI's > repository features, an altogether good thing given the global CDN powering > PyPI providing speed ups for a lot of people, however it did so by introducing > a new point of confusion and pain for both the end users and the authors. > > > Why Additional Repositories? > ---------------------------- > > The two common installer tools, pip and easy_install/setuptools, both support > the concept of additional locations to search for files to satisify the > installation requirements and have done so for many years. This means that > there is no need to "phase" in a new flag or concept and the solution to > installing a project from a repository other than PyPI will function regardless > of how old (within reason) the end user's installer is. Not only has this > concept existed in the Python tooling for some time, but it is a concept that > exists across languages and even extending to the OS level with OS package > tools almost universally using multiple repository support making it extremely > likely that someone is already familar with the concept. > > Additionally, the multiple repository approach is a concept that is useful > outside of the narrow scope of allowing projects which wish to be included on > the index portion of PyPI but do not wish to utilize the repository portion > of PyPI. This includes places where a company may wish to host a repository > that contains their internal packages or where a project may wish to have > multiple "channels" of releases, such as alpha, beta, release candidate, and > final release. > > Setting up an external repository is very simple, it can be achieved with > nothing more than a filesystem, some files to host, and any web server capable > of serving files and generating an automated index of directories (commonly > called "autoindex"). This can be as simple as: > > :: > > $ mkdir -p /var/www/index.example.com/ > $ mkdir -p /var/www/index.example.com/myproject/ > $ mv ~/myproject-1.0.tar.gz /var/www/index.example.com/myproject/ > $ twistd -n web --path /var/www/index.example.com/ > > > Using this additional location within pip is also simple and can be included > on a per invocation, per shell, or per user basis. The pip 6.0 will also > include the ability to configure this on a per virtual environment or per > machine basis as well. This can be as simple as: > > :: > > $ # As a CLI argument > $ pip install --extra-index-url https://index.example.com/ myproject > $ # As an environment variable > $ PIP_EXTRA_INDEX_URL=https://pypi.example.com/ pip install myproject > $ # With a configuration file > $ echo "[global]\nextra-index-url = https://pypi.example.com/" > ~/.pip/pip.conf > $ pip install myproject > > > Why Not PEP 438 or Similar? > --------------------------- > > While the additional search location support has existed in pip and setuptools > for quite some time support for PEP 438 has only existed in pip since the 1.4 > version, and still has yet to be implemented in setuptools. The design of > PEP 438 did mean that users still benefited for projects which did not require > external files even with older installers, however for projects which *did* > require external files, users are still silently being given either > potentionally unreliable or, even worse, unsafe files to download. This system > is also unique to Python as it arises out of the history of PyPI, this means > that it is almost certain that this concept will be foreign to most, if not all > users, until they encounter it while attempting to use the Python toolchain. > > Additionally, the classification system proposed by PEP 438 has, in practice, > turned out to be extremely confusing to end users, so much so that it is a > position of this PEP that the situation as it stands is completely untenable. > The common pattern for a user with this system is to attempt to install a > project possibly get an error message (or maybe not if the project ever > uploaded something to PyPI but later switched without removing old files), see > that the error message suggests ``--allow-external``, they reissue the command > adding that flag most likely getting another error message, see that this time > the error message suggests also adding ``--allow-unverified``, and again issue > the command a third time, this time finally getting the thing they wish to > install. > > This UX failure exists for several reasons. > > 1. If pip can locate files at all for a project on the Simple API it will > simply use that instead of attempting to locate more. This is generally the > right thing to do as attempting to locate more would erase a large part of > the benefit of PEP 438. This means that if a project *ever* uploaded > a file that matches what the user has requested for install that will be > used regardless of how old it is. > > 2. PEP 438 makes an implicit assumption that most projects would either upload > themselves to PyPI or would update themselves to directly linking to release > files. While a large number of projects *did* ultimately decide to upload > to PyPI, some of them did so only because the UX around what PEP 438 was so > bad that they felt forced to do so. More concerning however, is the fact > that very few projects have opted to directly and safely link to files and > instead they still simply link to pages which must be scraped in order to > find the actual files, thus rendering the safe variant > (``--allow-external``) largely useless. > > 3. Even if an author wishes to directly link to their files, doing so safely is > non-obvious. It requires the inclusion of a MD5 hash (for historical > reasons) in the hash of the URL. If they do not include this then their > files will be considered "unverified". > > 4. PEP 438 takes a security centric view and disallows any form of a global > opt in for unverified projects. While this is generally a good thing, it > creates extremely verbose and repetive command invocations such as: > > :: > > $ pip install --allow-external myproject --allow-unverified myproject myproject > $ pip install --allow-all-external --allow-unverified myproject myproject > > > Multiple Repository/Index Support > ================================= > > Installers SHOULD implement or continue to offer, the ability to point the > installer at multiple URL locations. The exact mechanisms for a user to > indicate they wish to use an additional location is left up to each indidivdual > implementation. > > Additionally the mechanism discovering an installation candidate when multiple > repositories are being used is also up to each individual implementation, > however once configured an implementation should not discourage, warn, or > otherwise cast a negative light upon the use of a repository simply because it > is not the default repository. > > Currently both pip and setuptools implement multiple repository support by > using the best installation candidate it can find from either repository, > essentially treating it as if it were one large repository. > > Installers SHOULD also implement some mechanism for removing or otherwise > disabling use of the default repository. The exact specifics of how that is > achieved is up to each indidivdual implementation. > > End users wishing to limit what files they pull from which repository can > simply use `devpi `_ to whitelist projects from > PyPI or another repository. > > > External Index Discovery > ======================== > > One of the problems with using an additional index is one of discovery. Users > will not generally be aware that an additional index is required at all much > less where that index can be found. Projects can attempt to convey this > information using their description on the PyPI page however that excludes > people who discover their project organically through ``pip search``. > > To support projects that wish to externally host their files and to enable > users to easily discover what additional indexes are required, PyPI will gain > the ability for projects to register external index URLs along with an > associated comment for each. These URLs will be made available on the simple > page however they will not be linked or provided in a form that older > installers will automatically search them. > > This ability will take the form of a ```` tag. The name of this tag must > be set to ``external-repository`` and the content will be a link to the location > of the external repository. An optional data-description attribute will convey > any comments or description that the author has provided. > > An example would look something like: > > :: > > > > > > When an external repository is added to a project, new uploads will no longer > be permitted to that project. However any existing files will simply be hidden > from the simple API and the web interface until all of the external repositories > are removed, in which case they will be visible again. PyPI MUST warn authors > if adding an external repository will hide files and that warning must persist > on any of the project management pages for that particular project. > > When an installer fetches the simple page for a project, if it finds this > additional meta-data and it cannot find any files for that project in it's > configured URLs then it should use this data to tell the user how to add one > or more of the additional URLs to search in. This message should include any > comments that the project has included to enable them to communicate to the > user and provide hints as to which URL they might want (e.g. if some are only > useful or compatible with certain platforms or situations). When the installer > has implemented the auto discovery mechanisms they should also deprecate any > of the mechanisms added for PEP 438 (such as ``--allow-external``) for removal > at the end of the deprecation period proposed by the PEP. > > This feature *must* be added to PyPI prior to starting the deprecation and > removal process for the implicit offsite hosting functionality. > > > Deprecation and Removal of Link Spidering > ========================================= > > A new hosting mode will be added to PyPI. This hosting mode will be called > ``pypi-only`` and will be in addition to the three that PEP 438 has already > given us which are ``pypi-explicit``, ``pypi-scrape``, ``pypi-scrape-crawl``. > This new hosting mode will modify a project's simple api page so that it only > lists the files which are directly hosted on PyPI and will not link to anything > else. > > Upon acceptance of this PEP and the addition of the ``pypi-only`` mode, all new > projects will be defaulted to the PyPI only mode and they will be locked to > this mode and unable to change this particular setting. ``pypi-only`` projects > will still be able to register external index URLs as described above - the > "pypi-only" refers only to the download links that are published directly on > PyPI. > > An email will then be sent out to all of the projects which are hosted only on > PyPI informing them that in one month their project will be automatically > converted to the ``pypi-only`` mode. A month after these emails have been sent > any of those projects which were emailed, which still are hosted only on PyPI > will have their mode set to ``pypi-only``. > > After that switch, an email will be sent to projects which rely on hosting > external to PyPI. This email will warn these projects that externally hosted > files have been deprecated on PyPI and that in 6 months from the time of that > email that all external links will be removed from the installer APIs. This > email *must* include instructions for converting their projects to be hosted > on PyPI and *must* include links to a script or package that will enable them > to enter their PyPI credentials and package name and have it automatically > download and re-host all of their files on PyPI. This email *must also* > include instructions for setting up their own index page and registering that > with PyPI, including the fact that they can use pythonhosted.org as a host > for an index page without requiring them to host any additional infrastructure > or purchase a TLS certificate. This email must also contain a link to the Terms > of Service for PyPI as many users may have signed up a long time ago and may > not recall what those terms are. > > Five months after the initial email, another email must be sent to any projects > still relying on external hosting. This email will include all of the same > information that the first email contained, except that the removal date will > be one month away instead of six. > > Finally a month later all projects will be switched to the ``pypi-only`` mode > and PyPI will be modified to remove the externally linked files functionality. > At this point in time any installers should finally remove any of the > deprecated PEP 438 functionality such as ``--allow-external`` and > ``--allow-unverified`` in pip. > > > Impact > ====== > > The largest impact of this is going to be projects where the maintainers are > no longer maintaining the project, for one reason or another. For these > projects it's unlikely that a maintainer will arrive to set the external index > metadata which would allow the auto discovery mechanism to find it. > > Looking at the numbers factoring out PIL (which has been special cased below) > the actual impact should be quite low, with it affecting just 3.8% of projects > which host any files only externally or 2.2% which have their latest version > hosted only externally. > > 6674 unique IP addresses have accessed the Simple API for these 3.8% of > projects in a single day (2014-09-30). Of those, 99.5% of them installed > something which could not be verified, and thus they were open to a Remote Code > Execution via a Man-In-The-Middle attack, while 7.9% installed something which > could be verified and only 0.4% only installed things which could be verified. > > > Projects Which Rely on Externally Hosted files > ---------------------------------------------- > > This is determined by crawling the simple index and looking for installable > files using a similar detection method as pip and setuptools use. The "latest" > version is determined using ``pkg_resources.parse_version`` sort order and it > is used to show whether or not the latest version is hosted externally or only > old versions are. > > ============ ======= ================ =================== ======= > \ PyPI External (old) External (latest) Total > ============ ======= ================ =================== ======= > **Safe** 43313 16 39 43368 > **Unsafe** 0 756 1092 1848 > **Total** 43313 772 1131 45216 > ============ ======= ================ =================== ======= > > > Top Externally Hosted Projects by Requests > ------------------------------------------ > > This is determined by looking at the number of requests the > ``/simple//`` page had gotten in a single day. The total number of > requests during that day was 10,623,831. > > ============================== ======== > Project Requests > ============================== ======== > PIL 63869 > Pygame 2681 > mysql-connector-python 1562 > pyodbc 724 > elementtree 635 > salesforce-python-toolkit 316 > wxPython 295 > PyXML 251 > RBTools 235 > python-graph-core 123 > cElementTree 121 > ============================== ======== > > > Top Externally Hosted Projects by Unique IPs > -------------------------------------------- > > This is determined by looking at the IP addresses of requests the > ``/simple//`` page had gotten in a single day. The total number of > unique IP addresses during that day was 124,604. > > ============================== ========== > Project Unique IPs > ============================== ========== > PIL 4553 > mysql-connector-python 462 > Pygame 202 > pyodbc 181 > elementtree 166 > wxPython 126 > RBTools 114 > PyXML 87 > salesforce-python-toolkit 76 > pyDes 76 > ============================== ========== > > > PIL > --- > > It's obvious from the numbers above that the vast bulk of the impact come from > the PIL project. On 2014-05-17 an email was sent to the contact for PIL > inquiring whether or not they would be willing to upload to PyPI. A response > has not been received as of yet (2014-10-03) nor has any change in the hosting > happened. Due to the popularity of PIL this PEP also proposes that during the > deprecation period that PyPI Administrators will set the PIL download URL as > the external index for that project. Allowing the users of PIL to take > advantage of the auto discovery mechanisms although the project has seemingly > become unmaintained. > > > Rejected Proposals > ================== > > Keep the current classification system but adjust the options > ------------------------------------------------------------- > > This PEP rejects several related proposals which attempt to fix some of the > usability problems with the current system but while still keeping the > general gist of PEP 438. > > This includes: > > * Default to allowing safely externally hosted files, but disallow unsafely > hosted. > * Default to disallowing safely externally hosted files with only a global > flag to enable them, but disallow unsafely hosted. > * Continue on the suggested path of PEP 438 and remove the option to unsafely > host externally but continue to allow the option to safely host externally. > > > These proposals are rejected because: > > * The classification system introduced in PEP 438 in an entirely unique concept > to PyPI which is not generically applicable even in the context of Python > packaging. Adding additional concepts comes at a cost. > > * The classification system itself is non-obvious to explain and to > pre-determine what classification of link a project will require entails > inspecting the project's ``/simple//`` page, and possibly any > URLs linked from that page. > > * The ability to host externally while still being linked for automatic > discovery is mostly a historic relic which causes a fair amount of pain and > complexity for little reward. > > * The installer's ability to optimize or clean up the user interface is limited > due to the nature of the implicit link scraping which would need to be done. > This extends to the ``--allow-*`` options as well as the inability to > determine if a link is expected to fail or not. > > * The mechanism paints a very broad brush when enabling an option, while PEP > 438 attempts to limit this with per package options. However a project that > has existed for an extended period of time may often times have several > different URLs listed in their simple index. It is not unsusual for at least > one of these to no longer be under control of the project. While an > unregistered domain will sit there relatively harmless most of the time, pip > will continue to attempt to install from it on every discovery phase. This > means that an attacker simply needs to look at projects which rely on unsafe > external URLs and register expired domains to attack users. > > Copyright > ========= > > This document has been placed in the public domain. > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From donald at stufft.io Fri Oct 3 14:02:05 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 3 Oct 2014 08:02:05 -0400 Subject: [Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting In-Reply-To: <20141003080629.GG7954@merlinux.eu> References: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> <20141003080629.GG7954@merlinux.eu> Message-ID: <5CD183C2-06B5-44B3-9104-566C7AF0BC76@stufft.io> > On Oct 3, 2014, at 4:06 AM, holger krekel wrote: > > Hi Donald, > > i could just only briefly glimpse over the new draft. I am still not in > favor of the PEP because it forces backard-incompatible changes and work > on various sides for not enough gain. Particularly end users will see > previously working commands now fail and if they run a new enough pip > version they get a hint on how to manually fix it. > > In the longer term you argue that people will appreciate reusing the concept > of dealing with multiple repositories as known with Linux Distros. > And that PEP470 would eventually simplify the user interface of pip and > the implementation of pypi a bit. > > I'll see next week if i can come up with suggestions how to reduce friction > introduced by the PEP while maintaining the benefits. And probably with > a few questions because i didn't understand all details yet i think. Mostly I think that the current situation is the worst possible implementation of handling offsite hosting that I can think of. I don't believe any person would design this system from scratch and the only reason it exists is because of lots of small decisions over the course of history. I think that the current implementation does a disservice to everyone involved. I think that it hurts end users because they end up most likely insecure, with tooling that cannot accurately report errors, and a horrible user interface. I think that it hurts authors, even those that want to rely on this feature, because it positions them in a case where it either implicitly breaks the expectations of end users who view PyPI mainly as a repository or it explicitly causes them pain in attempting to use and decipher the flags. It has been suggested that the changes pip made in implementing PEP 438 was done in an attempt to make not hosting on PyPI so painful that authors would be forced to host on PyPI. I think that backwards compatability is important, however I also think that it's important to break backwards compatability when it makes sense to do so. The current situation is such that attempting to preserve backwards compatability has made things worse for everyone involved whenever a project wishes to do anything but host their projects on PyPI. As far as simplication goes, I don't believe it simplifies the implementation of PyPI at all, it just shuffles things around and creates work on my part in order to get PyPI supporting the new stuff. It does however let installers become simpler and it enables installers to present accurate error information that actually helps determine the root cause of a failure instead of the current silent failure with a confusing error message model. I look forward to your suggestions, but I'm not hopeful. I've been thus far unable to determine a way to improve the current solution in a way that isn't just papering over one problem without solving the fundamental issue. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From marius at pov.lt Fri Oct 3 16:23:57 2014 From: marius at pov.lt (Marius Gedminas) Date: Fri, 3 Oct 2014 17:23:57 +0300 Subject: [Distutils] advice re: packaging tasks In-Reply-To: References: <20141002150805.GA28462@fridge.pov.lt> Message-ID: <20141003142357.GA27265@fridge.pov.lt> On Thu, Oct 02, 2014 at 09:15:54AM -0700, Chris Jerdonek wrote: > On Thu, Oct 2, 2014 at 8:08 AM, Marius Gedminas wrote: > > I wrote restview for a different purpose, but found it rather useful for > > discovering ReStructuredText problems that would make PyPI's fall back > > to plaintext rendering. > > > > If you run > > > > restview --long-description > > > > in a directory containing a setup.py, it'll invoke `python setup.py > > --long-description' and interpret it using the same docutils settings as > > PyPI, highlighting any errors. > > > > For a more automated (but perhaps less accurate) solution I pipe > > > > python setup.py --long-description | rst2html > /dev/null > > Yes, I had been familiar with this latter approach. It is documented here: > > https://docs.python.org/2/distutils/packageindex.html#pypi-package-display > > It did not work for me in my case though. I eventually found that my > use of URL fragments to link to sections elsewhere on the same page, > and using relative links (supported by GitHub, > https://github.com/blog/1395-relative-links-in-markup-files ) are what > broke things. Incidentally, this gotcha is why restview now has a --pypi-strict mode: https://github.com/mgedmin/restview/issues/18#issuecomment-54122181 (This is enabled by default for restview --long-description.) > Here is a link to a PyPI bug report regarding making it easier to find > such issues: > > https://bitbucket.org/pypa/pypi/issue/161/rest-formatting-fails-and-there-is-no-way > > If people are interested, I wrote a pandoc filter to convert such > links to things that will continue to work once uploaded to PyPI: > > https://gist.github.com/cjerdonek/76608610df43fd5b0fc3 ... > >> Lastly, as these setup-related tasks grow larger and more complicated, > >> I found it helped to break them out into a separate setup package that > >> sits alongside my project's main package library (and even adding > >> tests in some cases). Is this normal? Have other people run into > >> this? > > > > I'm not sure what you mean. Do you have any examples? > > I mean that if you are working on project MyProject with package > myproject inside the repo, you might find yourself adding ad hoc > custom code to setup.py. If this setup.py code starts to grow (a bit > like your Makefile may grow), it might make sense to move some of the > setup code to a package called something like myproject_setup > alongside myproject (which would be used only for setup tasks). And > if this setup code was sufficiently complicated, you might find > yourself wanting to add unit tests for some of it, so you might have > myproject_setup/tests (and even testing it as part of your automated > tests, etc). Right. I wanted to see some concrete examples of that code you find putting in your setup.py files. All I've seen before were bits that concatenate README.rst + CHANGES.rst into a long_description, or parse the version number from some source file in the name of DRY. Marius Gedminas -- A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing on usenet? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From ncoghlan at gmail.com Fri Oct 3 16:24:12 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 4 Oct 2014 00:24:12 +1000 Subject: [Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting In-Reply-To: <5CD183C2-06B5-44B3-9104-566C7AF0BC76@stufft.io> References: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> <20141003080629.GG7954@merlinux.eu> <5CD183C2-06B5-44B3-9104-566C7AF0BC76@stufft.io> Message-ID: On 3 October 2014 22:02, Donald Stufft wrote: > > As far as simplication goes, I don't believe it simplifies the implementation > of PyPI at all, it just shuffles things around and creates work on my part > in order to get PyPI supporting the new stuff. It does however let installers > become simpler and it enables installers to present accurate error information > that actually helps determine the root cause of a failure instead of the > current silent failure with a confusing error message model. > > I look forward to your suggestions, but I'm not hopeful. I've been thus far > unable to determine a way to improve the current solution in a way that isn't > just papering over one problem without solving the fundamental issue. Donald's perspective here matches my own. I'll be interested to hear alternative proposals, but they should aim to address at least the following user experience expectations: 1. Easily allow external hosting to "just work" when appropriately configured at the system, user or virtual environment level (pip already supports this at the user level, and will support it at the system and environment level in the next version). 2. Easily allow package authors to tell PyPI "my releases are hosted " and have that advertised in such a way that tools can clearly communicate it to users, without silently introducing unexpected dependencies on third party services. 3. Eliminate any and all references to the confusing "verifiable external" and "unverifiable external" distinction from the user experience (both when installing and when releasing packages). 4. The repository aspects of PyPI should become *just* the default package hosting location (i.e. the only one that is treated as opt-out rather than opt-in by most client tools in their default configuration). Aside from that aspect, hosting on PyPI should not otherwise provide an enhanced user experience over hosting your own package repository. 5. Do all of the above while providing default behaviour that is secure against most attackers below the nation state adversary level. In my view, the most debatable part of Donald's latest proposal would be the handling of projects that don't get updated to properly register an external URL before the link spidering support is removed from the client applications. That aspect should arguably include a step where the decision on whether or not to disable that support is based on *looking at the numbers again* before turning the feature off on the server, and perhaps also monitoring for user complaints for a period after it is first turned off, before the feature is removed from the clients. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From wichert at wiggy.net Fri Oct 3 16:29:25 2014 From: wichert at wiggy.net (Wichert Akkerman) Date: Fri, 3 Oct 2014 16:29:25 +0200 Subject: [Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting In-Reply-To: References: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> <20141003080629.GG7954@merlinux.eu> <5CD183C2-06B5-44B3-9104-566C7AF0BC76@stufft.io> Message-ID: > On 03 Oct 2014, at 16:24, Nick Coghlan wrote: > 2. Easily allow package authors to tell PyPI "my releases are hosted > " and have that advertised in such a way that tools can clearly > communicate it to users, without silently introducing unexpected > dependencies on third party services. I haven?t read the PEP, so this might be a stupid remark, but: is that needed, when a package author can also say something like ?add my repository to your system with pip ?add-repository ? ? Wichert. From marius at pov.lt Fri Oct 3 16:42:43 2014 From: marius at pov.lt (Marius Gedminas) Date: Fri, 3 Oct 2014 17:42:43 +0300 Subject: [Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting In-Reply-To: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> References: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> Message-ID: <20141003144243.GB27265@fridge.pov.lt> On Fri, Oct 03, 2014 at 02:05:50AM -0400, Donald Stufft wrote: > Using this additional location within pip is also simple and can be included > on a per invocation, per shell, or per user basis. The pip 6.0 will also > include the ability to configure this on a per virtual environment or per > machine basis as well. This can be as simple as: > > :: > > $ # As a CLI argument > $ pip install --extra-index-url https://index.example.com/ myproject > $ # As an environment variable > $ PIP_EXTRA_INDEX_URL=https://pypi.example.com/ pip install myproject > $ # With a configuration file > $ echo "[global]\nextra-index-url = https://pypi.example.com/" > ~/.pip/pip.conf > $ pip install myproject This is where I get a question: what do I do if package X wants an extra repository FOO, and package Y wants an extra repository BAR, and my project relies on both X and Y? I assume the --extra-index-url=URL argument to pip install can be repeated multiple times. It's less clear what to do about environment variables or config file settings. Do I specify space-separated URLs? Newline separated? An example would be good. Marius Gedminas -- Jim's Three Laws of Engineering: 1. F = ma 2. You can't solve a problem unless you know the answer 3. You can't push a rope -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From p.f.moore at gmail.com Fri Oct 3 16:44:22 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 3 Oct 2014 15:44:22 +0100 Subject: [Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting In-Reply-To: References: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> <20141003080629.GG7954@merlinux.eu> <5CD183C2-06B5-44B3-9104-566C7AF0BC76@stufft.io> Message-ID: On 3 October 2014 15:29, Wichert Akkerman wrote: > >> On 03 Oct 2014, at 16:24, Nick Coghlan wrote: >> 2. Easily allow package authors to tell PyPI "my releases are hosted >> " and have that advertised in such a way that tools can clearly >> communicate it to users, without silently introducing unexpected >> dependencies on third party services. > > I haven?t read the PEP, so this might be a stupid remark, but: is that needed, when a package author can also say something like ?add my repository to your system with pip ?add-repository ? ? The logic is that if I say pip install foo and foo is not hosted on PyPI, I get an error saying "cannot find foo". The quoted point is saying that we want a way for the author of foo to add metadata to PyPI that lets pip give a more helpful message: pip install foo ERROR: No downloads for package 'foo' found. foo is hosted at the following repositoties: Main repository - http://foo.example.com/simple/ Windows wheels - http://wheels.foo.example.com/simple/ Use --index-url to specify the repository you wish to use. (Or something like that...) Paul From graffatcolmingov at gmail.com Fri Oct 3 16:44:44 2014 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Fri, 3 Oct 2014 09:44:44 -0500 Subject: [Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting In-Reply-To: <20141003144243.GB27265@fridge.pov.lt> References: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> <20141003144243.GB27265@fridge.pov.lt> Message-ID: On Fri, Oct 3, 2014 at 9:42 AM, Marius Gedminas wrote: > On Fri, Oct 03, 2014 at 02:05:50AM -0400, Donald Stufft wrote: >> Using this additional location within pip is also simple and can be included >> on a per invocation, per shell, or per user basis. The pip 6.0 will also >> include the ability to configure this on a per virtual environment or per >> machine basis as well. This can be as simple as: >> >> :: >> >> $ # As a CLI argument >> $ pip install --extra-index-url https://index.example.com/ myproject >> $ # As an environment variable >> $ PIP_EXTRA_INDEX_URL=https://pypi.example.com/ pip install myproject >> $ # With a configuration file >> $ echo "[global]\nextra-index-url = https://pypi.example.com/" > ~/.pip/pip.conf >> $ pip install myproject > > This is where I get a question: what do I do if package X wants an > extra repository FOO, and package Y wants an extra repository BAR, and > my project relies on both X and Y? > > I assume the --extra-index-url=URL argument to pip install can be > repeated multiple times. It's less clear what to do about environment > variables or config file settings. Do I specify space-separated URLs? > Newline separated? > > An example would be good. > > Marius Gedminas I would assume something like $ PIP_EXTRA_INDEX_URL=https://pypi1.example.com/,https://pypi2.example.com/ pip install myproject $ pip install --extra-index-url=https://pypi1.example.com/ --extra-index-url=https://pypi2.example.com/ myproject From p.f.moore at gmail.com Fri Oct 3 16:52:24 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 3 Oct 2014 15:52:24 +0100 Subject: [Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting In-Reply-To: References: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> <20141003144243.GB27265@fridge.pov.lt> Message-ID: On 3 October 2014 15:44, Ian Cordasco wrote: > I would assume something like > > $ PIP_EXTRA_INDEX_URL=https://pypi1.example.com/,https://pypi2.example.com/ Separate with spaces. See https://pip.pypa.io/en/latest/user_guide.html#configuration for the details. Paul From chris.jerdonek at gmail.com Fri Oct 3 17:11:52 2014 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Fri, 3 Oct 2014 08:11:52 -0700 Subject: [Distutils] advice re: packaging tasks In-Reply-To: <20141003142357.GA27265@fridge.pov.lt> References: <20141002150805.GA28462@fridge.pov.lt> <20141003142357.GA27265@fridge.pov.lt> Message-ID: On Fri, Oct 3, 2014 at 7:23 AM, Marius Gedminas wrote: > On Thu, Oct 02, 2014 at 09:15:54AM -0700, Chris Jerdonek wrote: >> On Thu, Oct 2, 2014 at 8:08 AM, Marius Gedminas wrote: >> >> Lastly, as these setup-related tasks grow larger and more complicated, >> >> I found it helped to break them out into a separate setup package that >> >> sits alongside my project's main package library (and even adding >> >> tests in some cases). Is this normal? Have other people run into >> >> this? >> > >> > I'm not sure what you mean. Do you have any examples? >> >> I mean that if you are working on project MyProject with package >> myproject inside the repo, you might find yourself adding ad hoc >> custom code to setup.py. If this setup.py code starts to grow (a bit >> like your Makefile may grow), it might make sense to move some of the >> setup code to a package called something like myproject_setup >> alongside myproject (which would be used only for setup tasks). And >> if this setup code was sufficiently complicated, you might find >> yourself wanting to add unit tests for some of it, so you might have >> myproject_setup/tests (and even testing it as part of your automated >> tests, etc). > > Right. I wanted to see some concrete examples of that code you find > putting in your setup.py files. > > All I've seen before were bits that concatenate README.rst + CHANGES.rst > into a long_description, or parse the version number from some source > file in the name of DRY. In some older projects, I had a fair amount of code that essentially did what "check-manifest" does (but as part of running sdist). And yes, there was also code to do some pre-processing of README.md and to convert that with other files into a .rst long_description. In a newer project, I'm starting to add additional code to support running pandoc, which you can see here, for example: https://github.com/cjerdonek/open-rcv/blob/master/openrcv_setup/pandoc.py I use pandoc to convert files like README.md into both rst files for the long_description, and into HTML for local viewing (a bit similar to what restview does). I'm not sure what other tools people use for this, but I've been happy with pandoc, and it is quite versatile and powerful. I can point you to additional specific examples off-list if you want. --Chris From holger at merlinux.eu Fri Oct 3 20:28:00 2014 From: holger at merlinux.eu (holger krekel) Date: Fri, 3 Oct 2014 18:28:00 +0000 Subject: [Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting In-Reply-To: References: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> <20141003080629.GG7954@merlinux.eu> <5CD183C2-06B5-44B3-9104-566C7AF0BC76@stufft.io> Message-ID: <20141003182800.GH7954@merlinux.eu> On Sat, Oct 04, 2014 at 00:24 +1000, Nick Coghlan wrote: > On 3 October 2014 22:02, Donald Stufft wrote: > > > As far as simplication goes, I don't believe it simplifies the implementation > > of PyPI at all, it just shuffles things around and creates work on my part > > in order to get PyPI supporting the new stuff. It does however let installers > > become simpler and it enables installers to present accurate error information > > that actually helps determine the root cause of a failure instead of the > > current silent failure with a confusing error message model. > > > > I look forward to your suggestions, but I'm not hopeful. I've been thus far > > unable to determine a way to improve the current solution in a way that isn't > > just papering over one problem without solving the fundamental issue. > > Donald's perspective here matches my own. I don't see the "the fundamental issue" that PEP470 tries to solve. The first para of the abstract says it wants to substitute the existing mechanism for registering external indexes with another one. It doesn't say why. And it doesn't say why this can't be done in a backward compatible manner which would be preferable (i hope we agree there). And because the PEP doesn't precisely say what "fundamental issue" it solves it's a bit hard to present an alternative. If it's about focusing on "multi-repository operations" and simplifying installer UI it could be done with full backward compat: - add PyPI maintainer UI to add external indexes along with a message - change pip to disallow crawling to an external index it finds but rather present a message that you need to add the index manually to your installer invocation. (pip already finds external crawl URLs and it can also find the "new" ones - no need for any breakage). - tell all project maintainers which have "explicit file urls" that they need to move their release files to an offsite own external index (or to pypi itself) within N months. Then disable the file urls (after examination of how many people are effected) and remove related un-needed options in pip. Of course, i leave out some details but overall think it's pretty much doable. With this strategy, both old and new versions of pip wold work fine with the changed PyPI. It also wouldn't introduce very complicated transition phases or communication steps. I postpone other issues with respect to clarity and security of PEP/multi-repo operations to first get clarity on the backward compat issue and general strategy. best, holger P.S.: Nick, i think my rough draft above satisfies all of your points below, although they only partly relate to what we discuss in the PEP IMHO. > I'll be interested to hear alternative proposals, but they should aim > to address at least the following user experience expectations: > > 1. Easily allow external hosting to "just work" when appropriately > configured at the system, user or virtual environment level (pip > already supports this at the user level, and will support it at the > system and environment level in the next version). > > 2. Easily allow package authors to tell PyPI "my releases are hosted > " and have that advertised in such a way that tools can clearly > communicate it to users, without silently introducing unexpected > dependencies on third party services. > > 3. Eliminate any and all references to the confusing "verifiable > external" and "unverifiable external" distinction from the user > experience (both when installing and when releasing packages). > > 4. The repository aspects of PyPI should become *just* the default > package hosting location (i.e. the only one that is treated as opt-out > rather than opt-in by most client tools in their default > configuration). Aside from that aspect, hosting on PyPI should not > otherwise provide an enhanced user experience over hosting your own > package repository. > > 5. Do all of the above while providing default behaviour that is > secure against most attackers below the nation state adversary level. > > In my view, the most debatable part of Donald's latest proposal would > be the handling of projects that don't get updated to properly > register an external URL before the link spidering support is removed > from the client applications. That aspect should arguably include a > step where the decision on whether or not to disable that support is > based on *looking at the numbers again* before turning the feature off > on the server, and perhaps also monitoring for user complaints for a > period after it is first turned off, before the feature is removed > from the clients. > > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > From donald at stufft.io Fri Oct 3 20:43:47 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 3 Oct 2014 14:43:47 -0400 Subject: [Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting In-Reply-To: References: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> <20141003080629.GG7954@merlinux.eu> <5CD183C2-06B5-44B3-9104-566C7AF0BC76@stufft.io> Message-ID: > On Oct 3, 2014, at 10:29 AM, Wichert Akkerman wrote: > > >> On 03 Oct 2014, at 16:24, Nick Coghlan wrote: >> 2. Easily allow package authors to tell PyPI "my releases are hosted >> " and have that advertised in such a way that tools can clearly >> communicate it to users, without silently introducing unexpected >> dependencies on third party services. > > I haven?t read the PEP, so this might be a stupid remark, but: is that needed, when a package author can also say something like ?add my repository to your system with pip ?add-repository ? ? > > Wichert. So it?s not strictly required, and for pip versions less than 6.0 that?s essentially what will be happening. However providing that mechanism makes the discovery story a lot nicer. Instead of ``pip install foo`` coming back with a ?I can?t find any downloads? error, it can come back with a ?I can?t find any downloads, but here is the repositories that the author says they are using?. Basically it?s an affordance to make the UX of an external repository better. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From donald at stufft.io Fri Oct 3 20:49:05 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 3 Oct 2014 14:49:05 -0400 Subject: [Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting In-Reply-To: References: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> <20141003080629.GG7954@merlinux.eu> <5CD183C2-06B5-44B3-9104-566C7AF0BC76@stufft.io> Message-ID: > On Oct 3, 2014, at 10:44 AM, Paul Moore wrote: > > On 3 October 2014 15:29, Wichert Akkerman wrote: >> >>> On 03 Oct 2014, at 16:24, Nick Coghlan wrote: >>> 2. Easily allow package authors to tell PyPI "my releases are hosted >>> " and have that advertised in such a way that tools can clearly >>> communicate it to users, without silently introducing unexpected >>> dependencies on third party services. >> >> I haven?t read the PEP, so this might be a stupid remark, but: is that needed, when a package author can also say something like ?add my repository to your system with pip ?add-repository ? ? > > The logic is that if I say > > pip install foo > > and foo is not hosted on PyPI, I get an error saying "cannot find > foo". The quoted point is saying that we want a way for the author of > foo to add metadata to PyPI that lets pip give a more helpful message: > > pip install foo > ERROR: No downloads for package 'foo' found. > foo is hosted at the following repositoties: > Main repository - http://foo.example.com/simple/ > Windows wheels - http://wheels.foo.example.com/simple/ > Use --index-url to specify the repository you wish to use. > > (Or something like that?) Yes, the comments in particular were inspired by Egenix?s own repositories. I saw that they had different repositories for UCS2 and UCS4 and I thought that it would be awesome if pip could tell end users about both of those and give hem the information to choose between which ones were relevant. In this way it supports more things than the existing mechanisms support because the old things don?t allow any mechanism for selective addition which means binary distributions are hard if the filename doesn?t include enough information for pip/easy_install to actually select the proper download and you have to encode some of that information in the URL. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From donald at stufft.io Fri Oct 3 21:08:26 2014 From: donald at stufft.io (Donald Stufft) Date: Fri, 3 Oct 2014 15:08:26 -0400 Subject: [Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting In-Reply-To: <20141003182800.GH7954@merlinux.eu> References: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> <20141003080629.GG7954@merlinux.eu> <5CD183C2-06B5-44B3-9104-566C7AF0BC76@stufft.io> <20141003182800.GH7954@merlinux.eu> Message-ID: > On Oct 3, 2014, at 2:28 PM, holger krekel wrote: > > On Sat, Oct 04, 2014 at 00:24 +1000, Nick Coghlan wrote: >> On 3 October 2014 22:02, Donald Stufft wrote: >> >>> As far as simplication goes, I don't believe it simplifies the implementation >>> of PyPI at all, it just shuffles things around and creates work on my part >>> in order to get PyPI supporting the new stuff. It does however let installers >>> become simpler and it enables installers to present accurate error information >>> that actually helps determine the root cause of a failure instead of the >>> current silent failure with a confusing error message model. >>> >>> I look forward to your suggestions, but I'm not hopeful. I've been thus far >>> unable to determine a way to improve the current solution in a way that isn't >>> just papering over one problem without solving the fundamental issue. >> >> Donald's perspective here matches my own. > > I don't see the "the fundamental issue" that PEP470 tries to solve. > The first para of the abstract says it wants to substitute the existing > mechanism for registering external indexes with another one. It doesn't > say why. And it doesn't say why this can't be done in a backward > compatible manner which would be preferable (i hope we agree there). The fundamental issue is that PyPI is really two things, an index and a repository. Currently these two roles are blurred and that lack of distinction causes problems for both end users and authors and those problems create a certain animosity towards people not wanting to use PyPI as their repository. To this aim end users should be aware when they are installing things from a repository other than PyPI and they should also be aware when doing so is unsafe on the wire. PEP 438 solves this problem. End users opt in to using a repository other than PyPI. However It is my belief that the pain of doing so has outweighed the benefits of PEP 438. Thus PEP 470 attempts to "go back to the drawing board" and questions the mechanism for hosting on an alternative repository all together. > > And because the PEP doesn't precisely say what "fundamental issue" > it solves it's a bit hard to present an alternative. If it's about > focusing on "multi-repository operations" and simplifying installer UI > it could be done with full backward compat: > > - add PyPI maintainer UI to add external indexes along with a message Ok, this is part of PEP 470 too. > > - change pip to disallow crawling to an external index it finds > but rather present a message that you need to add the index > manually to your installer invocation. (pip already finds external > crawl URLs and it can also find the "new" ones - no need for > any breakage). I had thought of similar things, and my reasons for not using an and instead using a meta tag and for removing the old URLs instead of just making this in addition to is: 1. I don?t *want* users of older versions of pip/easy_install to implicitly be fetching these things, they should be able to opt in as well and indeed all the mechanisms exist in pip/easy_install for them to already do so. The only thing that doesn?t exist is the discovery mechanism. 2. This doesn?t actually prevent breakage, it just links the breakage to the version of pip/easy_install someone is using at the cost that people with older clients are implicitly fetching things, some of which may or may not be safe. Overall I think the goal of not breaking things is a good one, however PyPI isn?t a versioned thing where people can limit what version of things they run. It?s important just from a maintenance aspect to be able to deprecate and remove things over time. This will break things for people depending on those things of course, so it?s always a balancing act about deciding *when* exactly to remove something. I think that this is a good time to remove this particular thing because the core functionality of it?s replacement has existed for a long time, the actual use of the feature is quite low, and leaving it in presents an issue with usability and security. > > - tell all project maintainers which have "explicit file urls" > that they need to move their release files to an offsite > own external index (or to pypi itself) within N months. > Then disable the file urls (after examination of how many people > are effected) and remove related un-needed options in pip. This is still breakage for people using an older version of pip/easy_install, although a smaller set of things will break in this sense. > > Of course, i leave out some details but overall think it's pretty much > doable. With this strategy, both old and new versions of pip wold work > fine with the changed PyPI. It also wouldn't introduce very complicated > transition phases or communication steps. > > I postpone other issues with respect to clarity and security of > PEP/multi-repo operations to first get clarity on the backward compat > issue and general strategy. > > best, > holger > > P.S.: Nick, i think my rough draft above satisfies all of your points > below, although they only partly relate to what we discuss in the PEP > IMHO. > > >> I'll be interested to hear alternative proposals, but they should aim >> to address at least the following user experience expectations: >> >> 1. Easily allow external hosting to "just work" when appropriately >> configured at the system, user or virtual environment level (pip >> already supports this at the user level, and will support it at the >> system and environment level in the next version). >> >> 2. Easily allow package authors to tell PyPI "my releases are hosted >> " and have that advertised in such a way that tools can clearly >> communicate it to users, without silently introducing unexpected >> dependencies on third party services. >> >> 3. Eliminate any and all references to the confusing "verifiable >> external" and "unverifiable external" distinction from the user >> experience (both when installing and when releasing packages). >> >> 4. The repository aspects of PyPI should become *just* the default >> package hosting location (i.e. the only one that is treated as opt-out >> rather than opt-in by most client tools in their default >> configuration). Aside from that aspect, hosting on PyPI should not >> otherwise provide an enhanced user experience over hosting your own >> package repository. >> >> 5. Do all of the above while providing default behaviour that is >> secure against most attackers below the nation state adversary level. >> >> In my view, the most debatable part of Donald's latest proposal would >> be the handling of projects that don't get updated to properly >> register an external URL before the link spidering support is removed >> from the client applications. That aspect should arguably include a >> step where the decision on whether or not to disable that support is >> based on *looking at the numbers again* before turning the feature off >> on the server, and perhaps also monitoring for user complaints for a >> period after it is first turned off, before the feature is removed >> from the clients. >> >> Regards, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From ncoghlan at gmail.com Sat Oct 4 09:46:27 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 4 Oct 2014 17:46:27 +1000 Subject: [Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting In-Reply-To: References: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> <20141003080629.GG7954@merlinux.eu> <5CD183C2-06B5-44B3-9104-566C7AF0BC76@stufft.io> <20141003182800.GH7954@merlinux.eu> Message-ID: On 4 October 2014 05:08, Donald Stufft wrote: > 2. This doesn?t actually prevent breakage, it just links the breakage to the > version of pip/easy_install someone is using at the cost that people with > older clients are implicitly fetching things, some of which may or may not > be safe. > > Overall I think the goal of not breaking things is a good one, however PyPI > isn?t a versioned thing where people can limit what version of things they run. > It?s important just from a maintenance aspect to be able to deprecate and > remove things over time. This will break things for people depending on those > things of course, so it?s always a balancing act about deciding *when* exactly > to remove something. I think that this is a good time to remove this particular > thing because the core functionality of it?s replacement has existed for a long > time, the actual use of the feature is quite low, and leaving it in presents an > issue with usability and security. It occurred to me that it's potentially desirable to decouple the "stop advertising links for spidering from PyPI" step from the "stop supporting link spidering in the clients" step. My rationale is that the first is just about changing PyPI itself - more clearly splitting the "PyPI as index" and "PyPI as repository" roles. We can quantify that impact fairly clearly, and will have data to make informed decisions each step of the way. Removing the link spidering support from the *clients* is a potentially bigger deal, as it would impact anyone that was using link spidering *independently of PyPI*. We don't have any data on that, and it's a decision different clients may want to approach differently. So while PEP 470 would allow clients to *consider* dropping link spidering support (and any new clients would be free to never add it), it likely doesn't make sense for the PEP to commit any clients (including pip) to a particular time frame for dropping the feature. That would narrow the scope to just server side PyPI changes (with client updates to report the availability of external repositories being a quality of implementation issue rather than a hard requirement). Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Sat Oct 4 19:21:18 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 4 Oct 2014 13:21:18 -0400 Subject: [Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting In-Reply-To: References: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> <20141003080629.GG7954@merlinux.eu> <5CD183C2-06B5-44B3-9104-566C7AF0BC76@stufft.io> <20141003182800.GH7954@merlinux.eu> Message-ID: <912170E2-EBB7-4ACD-9532-4AC0C4187477@stufft.io> > On Oct 4, 2014, at 3:46 AM, Nick Coghlan wrote: > > On 4 October 2014 05:08, Donald Stufft wrote: >> 2. This doesn?t actually prevent breakage, it just links the breakage to the >> version of pip/easy_install someone is using at the cost that people with >> older clients are implicitly fetching things, some of which may or may not >> be safe. >> >> Overall I think the goal of not breaking things is a good one, however PyPI >> isn?t a versioned thing where people can limit what version of things they run. >> It?s important just from a maintenance aspect to be able to deprecate and >> remove things over time. This will break things for people depending on those >> things of course, so it?s always a balancing act about deciding *when* exactly >> to remove something. I think that this is a good time to remove this particular >> thing because the core functionality of it?s replacement has existed for a long >> time, the actual use of the feature is quite low, and leaving it in presents an >> issue with usability and security. > > It occurred to me that it's potentially desirable to decouple the > "stop advertising links for spidering from PyPI" step from the "stop > supporting link spidering in the clients" step. > > My rationale is that the first is just about changing PyPI itself - > more clearly splitting the "PyPI as index" and "PyPI as repository" > roles. We can quantify that impact fairly clearly, and will have data > to make informed decisions each step of the way. > > Removing the link spidering support from the *clients* is a > potentially bigger deal, as it would impact anyone that was using link > spidering *independently of PyPI*. We don't have any data on that, and > it's a decision different clients may want to approach differently. > > So while PEP 470 would allow clients to *consider* dropping link > spidering support (and any new clients would be free to never add it), > it likely doesn't make sense for the PEP to commit any clients > (including pip) to a particular time frame for dropping the feature. > That would narrow the scope to just server side PyPI changes (with > client updates to report the availability of external repositories > being a quality of implementation issue rather than a hard > requirement). Yea, I don?t think I included what the installers do in this PEP other than the parts specific to this PEP, so: 1. Implement multiple repository support. 2. Implement some mechanism for removing/disabling the default repository 3. Implement the discovery mechanism. 4. Deprecate / Remove PEP 438 I purposely don't give exact details how it should be done, as I think that each installer should decide how best to integrate that within their own UX. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From ncoghlan at gmail.com Sun Oct 5 04:06:11 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 5 Oct 2014 12:06:11 +1000 Subject: [Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting In-Reply-To: <912170E2-EBB7-4ACD-9532-4AC0C4187477@stufft.io> References: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> <20141003080629.GG7954@merlinux.eu> <5CD183C2-06B5-44B3-9104-566C7AF0BC76@stufft.io> <20141003182800.GH7954@merlinux.eu> <912170E2-EBB7-4ACD-9532-4AC0C4187477@stufft.io> Message-ID: On 5 October 2014 03:21, Donald Stufft wrote: >> On Oct 4, 2014, at 3:46 AM, Nick Coghlan wrote: >> So while PEP 470 would allow clients to *consider* dropping link >> spidering support (and any new clients would be free to never add it), >> it likely doesn't make sense for the PEP to commit any clients >> (including pip) to a particular time frame for dropping the feature. >> That would narrow the scope to just server side PyPI changes (with >> client updates to report the availability of external repositories >> being a quality of implementation issue rather than a hard >> requirement). > > Yea, I don?t think I included what the installers do in this PEP other than > the parts specific to this PEP, so: > > 1. Implement multiple repository support. > 2. Implement some mechanism for removing/disabling the default repository > 3. Implement the discovery mechanism. > 4. Deprecate / Remove PEP 438 > > I purposely don't give exact details how it should be done, as I think that > each installer should decide how best to integrate that within their own > UX. I think it's worth spelling out that list of updated client expectations clearly in the PEP, with step 4 explicitly flagged as optional. If any given client wants to continue supporting PEP 438 for use with private indexes, I think that's fine. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Sun Oct 5 04:16:07 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 4 Oct 2014 22:16:07 -0400 Subject: [Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting In-Reply-To: References: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> <20141003080629.GG7954@merlinux.eu> <5CD183C2-06B5-44B3-9104-566C7AF0BC76@stufft.io> <20141003182800.GH7954@merlinux.eu> <912170E2-EBB7-4ACD-9532-4AC0C4187477@stufft.io> Message-ID: <722777EB-9170-4A2B-99D4-FB374A9C082B@stufft.io> > On Oct 4, 2014, at 10:06 PM, Nick Coghlan wrote: > > On 5 October 2014 03:21, Donald Stufft wrote: >>> On Oct 4, 2014, at 3:46 AM, Nick Coghlan wrote: >>> So while PEP 470 would allow clients to *consider* dropping link >>> spidering support (and any new clients would be free to never add it), >>> it likely doesn't make sense for the PEP to commit any clients >>> (including pip) to a particular time frame for dropping the feature. >>> That would narrow the scope to just server side PyPI changes (with >>> client updates to report the availability of external repositories >>> being a quality of implementation issue rather than a hard >>> requirement). >> >> Yea, I don?t think I included what the installers do in this PEP other than >> the parts specific to this PEP, so: >> >> 1. Implement multiple repository support. >> 2. Implement some mechanism for removing/disabling the default repository >> 3. Implement the discovery mechanism. >> 4. Deprecate / Remove PEP 438 >> >> I purposely don't give exact details how it should be done, as I think that >> each installer should decide how best to integrate that within their own >> UX. > > I think it's worth spelling out that list of updated client > expectations clearly in the PEP, with step 4 explicitly flagged as > optional. If any given client wants to continue supporting PEP 438 for > use with private indexes, I think that's fine. > > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia Updated my local copy to have this, it?ll be included in my next draft. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From j.j.molenaar at gmail.com Mon Oct 6 12:39:57 2014 From: j.j.molenaar at gmail.com (Joost Molenaar) Date: Mon, 6 Oct 2014 12:39:57 +0200 Subject: [Distutils] pypi simple page for setuptools returning 11034 links Message-ID: Hi all, When I run `pip list --outdated` on my raspberry pi, pip seems to hang on setuptools. Investigating a bit, I found that currently, there are 11034 links being returned [1], weighing 1379 KiB. Many of these are links to issue pages on github and bitbucket, also, many of these links are duplicated 20-25 times. Is this normal/expected? Is it a bug, if so, should I file it somewhere? Regards, Joost Molenaar [1] https://pypi.python.org/simple/setuptools/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Mon Oct 6 12:59:02 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 6 Oct 2014 06:59:02 -0400 Subject: [Distutils] pypi simple page for setuptools returning 11034 links In-Reply-To: References: Message-ID: <66326266-8895-454F-ADA5-58CF65C955C3@stufft.io> > On Oct 6, 2014, at 6:39 AM, Joost Molenaar wrote: > > Hi all, > > When I run `pip list --outdated` on my raspberry pi, pip seems to hang on setuptools. > Investigating a bit, I found that currently, there are 11034 links being returned [1], > weighing 1379 KiB. Many of these are links to issue pages on github and bitbucket, > also, many of these links are duplicated 20-25 times. > > Is this normal/expected? Is it a bug, if so, should I file it somewhere? > > Regards, > > Joost Molenaar > > [1] https://pypi.python.org/simple/setuptools/ > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig It?s normal in that it?s what PyPI has always done and changing it requires going through a process. It just so happens that I?ve started that process (see PEP 470!) and hopefully this will change in the near future :) --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Oct 6 13:17:45 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 6 Oct 2014 12:17:45 +0100 Subject: [Distutils] pypi simple page for setuptools returning 11034 links In-Reply-To: <66326266-8895-454F-ADA5-58CF65C955C3@stufft.io> References: <66326266-8895-454F-ADA5-58CF65C955C3@stufft.io> Message-ID: On 6 October 2014 11:59, Donald Stufft wrote: > On Oct 6, 2014, at 6:39 AM, Joost Molenaar wrote: >> When I run `pip list --outdated` on my raspberry pi, pip seems to hang on >> setuptools. Investigating a bit, I found that currently, there are 11034 links being >> returned [1], weighing 1379 KiB. Many of these are links to issue pages on github and >> bitbucket, also, many of these links are duplicated 20-25 times. >> >> Is this normal/expected? Is it a bug, if so, should I file it somewhere? > > It?s normal in that it?s what PyPI has always done and changing it requires > going through a process. > > It just so happens that I?ve started that process (see PEP 470!) and > hopefully this will change in the near future :) Having said that, I find the PyPI page for setuptools (which is the long_description field, which is where all those links come from) to be annoyingly long and unhelpful. Can I suggest that it would be helpful if setuptools removed some of the ancient history from the PyPI page? Possibly even replace the huge changelog with a link to the changelog file, rather than having it all inline? That would make the PyPI page more readable and usable, and as a fortuitous side-effect, remove all the useless links from the pypi simple page. (You could file a request like that as a bug report on the setuptools tracker). Paul From g.brandl at gmx.net Mon Oct 6 14:00:39 2014 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 06 Oct 2014 14:00:39 +0200 Subject: [Distutils] pypi simple page for setuptools returning 11034 links In-Reply-To: References: <66326266-8895-454F-ADA5-58CF65C955C3@stufft.io> Message-ID: On 10/06/2014 01:17 PM, Paul Moore wrote: > On 6 October 2014 11:59, Donald Stufft wrote: >> On Oct 6, 2014, at 6:39 AM, Joost Molenaar wrote: >>> When I run `pip list --outdated` on my raspberry pi, pip seems to hang on >>> setuptools. Investigating a bit, I found that currently, there are 11034 links being >>> returned [1], weighing 1379 KiB. Many of these are links to issue pages on github and >>> bitbucket, also, many of these links are duplicated 20-25 times. >>> >>> Is this normal/expected? Is it a bug, if so, should I file it somewhere? >> >> It?s normal in that it?s what PyPI has always done and changing it requires >> going through a process. >> >> It just so happens that I?ve started that process (see PEP 470!) and >> hopefully this will change in the near future :) > > Having said that, I find the PyPI page for setuptools (which is the > long_description field, which is where all those links come from) to > be annoyingly long and unhelpful. > > Can I suggest that it would be helpful if setuptools removed some of > the ancient history from the PyPI page? Possibly even replace the huge > changelog with a link to the changelog file, rather than having it all > inline? That would make the PyPI page more readable and usable, and as > a fortuitous side-effect, remove all the useless links from the pypi > simple page. Indeed. The table of contents entries alone for the changelog takes up four screen pages. Georg From p.f.moore at gmail.com Mon Oct 6 14:15:37 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 6 Oct 2014 13:15:37 +0100 Subject: [Distutils] pypi simple page for setuptools returning 11034 links In-Reply-To: References: <66326266-8895-454F-ADA5-58CF65C955C3@stufft.io> Message-ID: On 6 October 2014 13:00, Georg Brandl wrote: > ndeed. The table of contents entries alone for the changelog takes up > four screen pages. https://bitbucket.org/pypa/setuptools/issue/265/pypi-page-and-simple-index-for-setuptools From holger at merlinux.eu Tue Oct 7 12:09:25 2014 From: holger at merlinux.eu (holger krekel) Date: Tue, 7 Oct 2014 10:09:25 +0000 Subject: [Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting In-Reply-To: References: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> <20141003080629.GG7954@merlinux.eu> <5CD183C2-06B5-44B3-9104-566C7AF0BC76@stufft.io> <20141003182800.GH7954@merlinux.eu> Message-ID: <20141007100925.GM7954@merlinux.eu> On Fri, Oct 03, 2014 at 15:08 -0400, Donald Stufft wrote: > > On Oct 3, 2014, at 2:28 PM, holger krekel wrote: > > > > On Sat, Oct 04, 2014 at 00:24 +1000, Nick Coghlan wrote: > >> On 3 October 2014 22:02, Donald Stufft wrote: > >> > >>> As far as simplication goes, I don't believe it simplifies the implementation > >>> of PyPI at all, it just shuffles things around and creates work on my part > >>> in order to get PyPI supporting the new stuff. It does however let installers > >>> become simpler and it enables installers to present accurate error information > >>> that actually helps determine the root cause of a failure instead of the > >>> current silent failure with a confusing error message model. > >>> > >>> I look forward to your suggestions, but I'm not hopeful. I've been thus far > >>> unable to determine a way to improve the current solution in a way that isn't > >>> just papering over one problem without solving the fundamental issue. > >> > >> Donald's perspective here matches my own. > > > > I don't see the "the fundamental issue" that PEP470 tries to solve. > > The first para of the abstract says it wants to substitute the existing > > mechanism for registering external indexes with another one. It doesn't > > say why. And it doesn't say why this can't be done in a backward > > compatible manner which would be preferable (i hope we agree there). > > The fundamental issue is that PyPI is really two things, an index and a > repository. Currently these two roles are blurred and that lack of distinction > causes problems for both end users and authors and those problems create a > certain animosity towards people not wanting to use PyPI as their repository. > To this aim end users should be aware when they are installing things from > a repository other than PyPI and they should also be aware when doing so > is unsafe on the wire. > > PEP 438 solves this problem. End users opt in to using a repository other than > PyPI. However It is my belief that the pain of doing so has outweighed the > benefits of PEP 438. Well, the main benefit of PEP438 was that it removed random crawling for some 90% of the packages on the package index, speeding up and making installs more reliable. And it did that without breaking backward compatibility. And I think PEP470 could achieve its goals this way too. > Thus PEP 470 attempts to "go back to the drawing board" > and questions the mechanism for hosting on an alternative repository all > together. > > > > > And because the PEP doesn't precisely say what "fundamental issue" > > it solves it's a bit hard to present an alternative. If it's about > > focusing on "multi-repository operations" and simplifying installer UI > > it could be done with full backward compat: > > > > - add PyPI maintainer UI to add external indexes along with a message > > Ok, this is part of PEP 470 too. > > > > > - change pip to disallow crawling to an external index it finds > > but rather present a message that you need to add the index > > manually to your installer invocation. (pip already finds external > > crawl URLs and it can also find the "new" ones - no need for > > any breakage). > > I had thought of similar things, and my reasons for not using an href> and instead using a meta tag and for removing the old URLs > instead of just making this in addition to is: > > 1. I don?t *want* users of older versions of pip/easy_install to > implicitly be fetching these things, they should be able to opt in as > well and indeed all the mechanisms exist in pip/easy_install for them > to already do so. The only thing that doesn?t exist is the discovery > mechanism. I think it's better to generally avoid deliberately breaking things. Things break enough even when we don't intend them to. IOW, Pypi should IMO aim to preserve working with as many client side scenarios as possible -- while adding things and improving for newer versions of clients. > 2. This doesn?t actually prevent breakage, it just links the breakage > to the version of pip/easy_install someone is using at the cost that > people with older clients are implicitly fetching things, some of > which may or may not be safe. I am not sure i follow here, sorry. There are two things the PEP does: 1. remove "registered verified external links" 2. support recording external indexes for a project The first could be done without breakage except for the users and maintainers of that feature -- i take it we are still talking about just a few thousand client side uses and 60 project maintainers, right? The second could be done without breakage alltogether i think: at one time all external urls are auto-registered as external indexes and they are presented on the simple page with some meta information that does not confuse older pips/easy_installs. Newer pips/easy_installs can then provide nice error messages. Older pips can continue to use the PEP438 options. And easy install can continue to work. > Overall I think the goal of not breaking things is a good one, however PyPI > isn?t a versioned thing where people can limit what version of things they run. > It?s important just from a maintenance aspect to be able to deprecate and > remove things over time. This will break things for people depending on those > things of course, so it?s always a balancing act about deciding *when* exactly > to remove something. I think that this is a good time to remove this particular > thing because the core functionality of it?s replacement has existed for a long > time, the actual use of the feature is quite low, and leaving it in presents an > issue with usability and security. I agree that removing features and functionality is a good thing. But i maintain PEP470 could do it without breaking things. holger > > > > - tell all project maintainers which have "explicit file urls" > > that they need to move their release files to an offsite > > own external index (or to pypi itself) within N months. > > Then disable the file urls (after examination of how many people > > are effected) and remove related un-needed options in pip. > > This is still breakage for people using an older version of pip/easy_install, > although a smaller set of things will break in this sense. > > > > > Of course, i leave out some details but overall think it's pretty much > > doable. With this strategy, both old and new versions of pip wold work > > fine with the changed PyPI. It also wouldn't introduce very complicated > > transition phases or communication steps. > > > > I postpone other issues with respect to clarity and security of > > PEP/multi-repo operations to first get clarity on the backward compat > > issue and general strategy. > > > > best, > > holger > > > > P.S.: Nick, i think my rough draft above satisfies all of your points > > below, although they only partly relate to what we discuss in the PEP > > IMHO. > > > > > >> I'll be interested to hear alternative proposals, but they should aim > >> to address at least the following user experience expectations: > >> > >> 1. Easily allow external hosting to "just work" when appropriately > >> configured at the system, user or virtual environment level (pip > >> already supports this at the user level, and will support it at the > >> system and environment level in the next version). > >> > >> 2. Easily allow package authors to tell PyPI "my releases are hosted > >> " and have that advertised in such a way that tools can clearly > >> communicate it to users, without silently introducing unexpected > >> dependencies on third party services. > >> > >> 3. Eliminate any and all references to the confusing "verifiable > >> external" and "unverifiable external" distinction from the user > >> experience (both when installing and when releasing packages). > >> > >> 4. The repository aspects of PyPI should become *just* the default > >> package hosting location (i.e. the only one that is treated as opt-out > >> rather than opt-in by most client tools in their default > >> configuration). Aside from that aspect, hosting on PyPI should not > >> otherwise provide an enhanced user experience over hosting your own > >> package repository. > >> > >> 5. Do all of the above while providing default behaviour that is > >> secure against most attackers below the nation state adversary level. > >> > >> In my view, the most debatable part of Donald's latest proposal would > >> be the handling of projects that don't get updated to properly > >> register an external URL before the link spidering support is removed > >> from the client applications. That aspect should arguably include a > >> step where the decision on whether or not to disable that support is > >> based on *looking at the numbers again* before turning the feature off > >> on the server, and perhaps also monitoring for user complaints for a > >> period after it is first turned off, before the feature is removed > >> from the clients. > >> > >> Regards, > >> Nick. > >> > >> -- > >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > From mal at egenix.com Tue Oct 7 12:40:02 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 07 Oct 2014 12:40:02 +0200 Subject: [Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting In-Reply-To: <20141007100925.GM7954@merlinux.eu> References: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> <20141003080629.GG7954@merlinux.eu> <5CD183C2-06B5-44B3-9104-566C7AF0BC76@stufft.io> <20141003182800.GH7954@merlinux.eu> <20141007100925.GM7954@merlinux.eu> Message-ID: <5433C302.6080702@egenix.com> FWIW: I support Holger's request to introduce multi repository support *without* breaking existing setups. Simply add the possibility for authors to register external indexes, have pip, setuptools, et al. crawl these in addition to what's up on the PyPI package page (using the logic that has existed in these tools for years) and then let the author decide whether they want to remove existing downloads from PyPI or not. This allows for older installations to continue working, while also (optionally) supporting a setup which does not use PyPI for hosting at all. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From p.f.moore at gmail.com Tue Oct 7 12:40:26 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 7 Oct 2014 11:40:26 +0100 Subject: [Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting In-Reply-To: <20141007100925.GM7954@merlinux.eu> References: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> <20141003080629.GG7954@merlinux.eu> <5CD183C2-06B5-44B3-9104-566C7AF0BC76@stufft.io> <20141003182800.GH7954@merlinux.eu> <20141007100925.GM7954@merlinux.eu> Message-ID: On 7 October 2014 11:09, holger krekel wrote: > Well, the main benefit of PEP438 was that it removed random crawling for > some 90% of the packages on the package index, speeding up and making > installs more reliable. And it did that without breaking backward > compatibility. The setuptools index page is 1.4MB in size. Most of that can be ignored, but it still has to be downloaded and parsed. Whether the data that setuptools includes in its long_description is useful is arguable, but irrelevant - the fact is that as things stand, it is there and it causes issues. PEP 470 would result in all of the unneeded entries in the simple index for setuptools being removed, which avoids the need for client tools (and I'm not talking just about pip here, but also about one-off scripts, which is the sort of thing I write a lot) to trawl through all of that data. And it does so without the setuptools project having to change how it writes its PyPI page (i.e., the project long_description). Arguably, that's equally a way of avoiding breaking backward compatibility... > The second could be done without breakage alltogether i think: at one > time all external urls are auto-registered as external indexes > and they are presented on the simple page with some meta information > that does not confuse older pips/easy_installs. Newer pips/easy_installs > can then provide nice error messages. Older pips can continue to use > the PEP438 options. And easy install can continue to work. Setuptools has 255 internal links to files hosted on PyPI. And about 11,000 other links. (I just checked that 3 times, as I couldn't believe it, but it *seems* to be right :-(). Removing duplicates, 337 unique links. Are you suggesting pip presents all of those as possible external indexes? I'm sure you can argue that setuptools has (badly!) misused the link-handling support in PyPI. And that it's a one-off special case. But how do we document to projects that they shouldn't do things like this? How do we even define what "things like this" are? Don't include links in your project description unless they are external indexes? Paul. From holger at merlinux.eu Tue Oct 7 13:03:12 2014 From: holger at merlinux.eu (holger krekel) Date: Tue, 7 Oct 2014 11:03:12 +0000 Subject: [Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting In-Reply-To: References: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> <20141003080629.GG7954@merlinux.eu> <5CD183C2-06B5-44B3-9104-566C7AF0BC76@stufft.io> <20141003182800.GH7954@merlinux.eu> <20141007100925.GM7954@merlinux.eu> Message-ID: <20141007110312.GN7954@merlinux.eu> On Tue, Oct 07, 2014 at 11:40 +0100, Paul Moore wrote: > On 7 October 2014 11:09, holger krekel wrote: > > Well, the main benefit of PEP438 was that it removed random crawling for > > some 90% of the packages on the package index, speeding up and making > > installs more reliable. And it did that without breaking backward > > compatibility. > > The setuptools index page is 1.4MB in size. Most of that can be > ignored, but it still has to be downloaded and parsed. Whether the > data that setuptools includes in its long_description is useful is > arguable, but irrelevant - the fact is that as things stand, it is > there and it causes issues. > > PEP 470 would result in all of the unneeded entries in the simple > index for setuptools being removed, which avoids the need for client > tools (and I'm not talking just about pip here, but also about one-off > scripts, which is the sort of thing I write a lot) to trawl through > all of that data. And it does so without the setuptools project having > to change how it writes its PyPI page (i.e., the project > long_description). > Arguably, that's equally a way of avoiding breaking backward compatibility... > > > The second could be done without breakage alltogether i think: at one > > time all external urls are auto-registered as external indexes > > and they are presented on the simple page with some meta information > > that does not confuse older pips/easy_installs. Newer pips/easy_installs > > can then provide nice error messages. Older pips can continue to use > > the PEP438 options. And easy install can continue to work. > > Setuptools has 255 internal links to files hosted on PyPI. And about > 11,000 other links. (I just checked that 3 times, as I couldn't > believe it, but it *seems* to be right :-(). Removing duplicates, 337 > unique links. Are you suggesting pip presents all of those as possible > external indexes? No, i effectively suggest that PyPI would present just 2 index links, those which currently are attributed as rel={download,homepage}. Those two index links would be put into the new "extra indexes field" on pypi with a note like "the following indexes were extracted from old release data" which newer pip versions could present to the user. For older pip/easy_installs things would just continue to work but they'd get a shorter setuptools simple page. best, holger > I'm sure you can argue that setuptools has (badly!) misused the > link-handling support in PyPI. And that it's a one-off special case. > But how do we document to projects that they shouldn't do things like > this? How do we even define what "things like this" are? Don't include > links in your project description unless they are external indexes? > > Paul. > From donald at stufft.io Tue Oct 7 14:00:41 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 7 Oct 2014 08:00:41 -0400 Subject: [Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting In-Reply-To: <20141007100925.GM7954@merlinux.eu> References: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> <20141003080629.GG7954@merlinux.eu> <5CD183C2-06B5-44B3-9104-566C7AF0BC76@stufft.io> <20141003182800.GH7954@merlinux.eu> <20141007100925.GM7954@merlinux.eu> Message-ID: <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> > On Oct 7, 2014, at 6:09 AM, holger krekel wrote: > > On Fri, Oct 03, 2014 at 15:08 -0400, Donald Stufft wrote: >>> On Oct 3, 2014, at 2:28 PM, holger krekel wrote: >>> >>> On Sat, Oct 04, 2014 at 00:24 +1000, Nick Coghlan wrote: >>>> On 3 October 2014 22:02, Donald Stufft wrote: >>>> >>>>> As far as simplication goes, I don't believe it simplifies the implementation >>>>> of PyPI at all, it just shuffles things around and creates work on my part >>>>> in order to get PyPI supporting the new stuff. It does however let installers >>>>> become simpler and it enables installers to present accurate error information >>>>> that actually helps determine the root cause of a failure instead of the >>>>> current silent failure with a confusing error message model. >>>>> >>>>> I look forward to your suggestions, but I'm not hopeful. I've been thus far >>>>> unable to determine a way to improve the current solution in a way that isn't >>>>> just papering over one problem without solving the fundamental issue. >>>> >>>> Donald's perspective here matches my own. >>> >>> I don't see the "the fundamental issue" that PEP470 tries to solve. >>> The first para of the abstract says it wants to substitute the existing >>> mechanism for registering external indexes with another one. It doesn't >>> say why. And it doesn't say why this can't be done in a backward >>> compatible manner which would be preferable (i hope we agree there). >> >> The fundamental issue is that PyPI is really two things, an index and a >> repository. Currently these two roles are blurred and that lack of distinction >> causes problems for both end users and authors and those problems create a >> certain animosity towards people not wanting to use PyPI as their repository. >> To this aim end users should be aware when they are installing things from >> a repository other than PyPI and they should also be aware when doing so >> is unsafe on the wire. >> >> PEP 438 solves this problem. End users opt in to using a repository other than >> PyPI. However It is my belief that the pain of doing so has outweighed the >> benefits of PEP 438. > > Well, the main benefit of PEP438 was that it removed random crawling for > some 90% of the packages on the package index, speeding up and making > installs more reliable. And it did that without breaking backward > compatibility. And I think PEP470 could achieve its goals this way too. Sorry, I mean the main benefit with regards to projects that are hosted externally. PEP 438 had tremendous benefit for cleaning up a ton of projects which were not hosted externally and had links which existed for nothing more than to slow things down and make things unsafe. > >> Thus PEP 470 attempts to "go back to the drawing board" >> and questions the mechanism for hosting on an alternative repository all >> together. >> >>> >>> And because the PEP doesn't precisely say what "fundamental issue" >>> it solves it's a bit hard to present an alternative. If it's about >>> focusing on "multi-repository operations" and simplifying installer UI >>> it could be done with full backward compat: >>> >>> - add PyPI maintainer UI to add external indexes along with a message >> >> Ok, this is part of PEP 470 too. >> >>> >>> - change pip to disallow crawling to an external index it finds >>> but rather present a message that you need to add the index >>> manually to your installer invocation. (pip already finds external >>> crawl URLs and it can also find the "new" ones - no need for >>> any breakage). >> >> I had thought of similar things, and my reasons for not using an > href> and instead using a meta tag and for removing the old URLs >> instead of just making this in addition to is: >> >> 1. I don?t *want* users of older versions of pip/easy_install to >> implicitly be fetching these things, they should be able to opt in as >> well and indeed all the mechanisms exist in pip/easy_install for them >> to already do so. The only thing that doesn?t exist is the discovery >> mechanism. > > I think it's better to generally avoid deliberately breaking things. > Things break enough even when we don't intend them to. > > IOW, Pypi should IMO aim to preserve working with as many client side > scenarios as possible -- while adding things and improving for newer > versions of clients. And here I think is where the crux of our disagreement lies I think. I think that PyPI should preserve working with as many client side scenarios as possible, except where there is good reason to do so. I believe the fact that the vast bulk of the cases we?d be breaking are people who are silently, and often unknowingly, being directed to download some code over unauthenticated channels is a very good reason to break those cases. Especially given the fact that there is a fairly trivial work around for people who want to restore that behavior. In a way this is similar to switching Python to enforcing TLS verification by default, which afaik Guido has blessed even for 2.7 assuming that there is a sane way to restore the default behavior and configure it. > >> 2. This doesn?t actually prevent breakage, it just links the breakage >> to the version of pip/easy_install someone is using at the cost that >> people with older clients are implicitly fetching things, some of >> which may or may not be safe. > > I am not sure i follow here, sorry. There are two things the PEP does: > > 1. remove "registered verified external links" > > 2. support recording external indexes for a project > > The first could be done without breakage except for the users and maintainers > of that feature -- i take it we are still talking about just a few thousand > client side uses and 60 project maintainers, right? > > The second could be done without breakage alltogether i think: at one > time all external urls are auto-registered as external indexes > and they are presented on the simple page with some meta information > that does not confuse older pips/easy_installs. Newer pips/easy_installs > can then provide nice error messages. Older pips can continue to use > the PEP438 options. And easy install can continue to work. > >> Overall I think the goal of not breaking things is a good one, however PyPI >> isn?t a versioned thing where people can limit what version of things they run. >> It?s important just from a maintenance aspect to be able to deprecate and >> remove things over time. This will break things for people depending on those >> things of course, so it?s always a balancing act about deciding *when* exactly >> to remove something. I think that this is a good time to remove this particular >> thing because the core functionality of it?s replacement has existed for a long >> time, the actual use of the feature is quite low, and leaving it in presents an >> issue with usability and security. > > I agree that removing features and functionality is a good thing. > But i maintain PEP470 could do it without breaking things. > It absolutely *could*, but as described above, I think it?s a better idea to break things in this case. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From donald at stufft.io Tue Oct 7 14:04:15 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 7 Oct 2014 08:04:15 -0400 Subject: [Distutils] PEP 470, round 4 - Using Multi Repository Support for External to PyPI Package File Hosting In-Reply-To: <20141007110312.GN7954@merlinux.eu> References: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> <20141003080629.GG7954@merlinux.eu> <5CD183C2-06B5-44B3-9104-566C7AF0BC76@stufft.io> <20141003182800.GH7954@merlinux.eu> <20141007100925.GM7954@merlinux.eu> <20141007110312.GN7954@merlinux.eu> Message-ID: <626CEBB5-780E-4A09-8040-DFF23CCBCDC9@stufft.io> > On Oct 7, 2014, at 7:03 AM, holger krekel wrote: > > On Tue, Oct 07, 2014 at 11:40 +0100, Paul Moore wrote: >> On 7 October 2014 11:09, holger krekel wrote: >>> Well, the main benefit of PEP438 was that it removed random crawling for >>> some 90% of the packages on the package index, speeding up and making >>> installs more reliable. And it did that without breaking backward >>> compatibility. >> >> The setuptools index page is 1.4MB in size. Most of that can be >> ignored, but it still has to be downloaded and parsed. Whether the >> data that setuptools includes in its long_description is useful is >> arguable, but irrelevant - the fact is that as things stand, it is >> there and it causes issues. >> >> PEP 470 would result in all of the unneeded entries in the simple >> index for setuptools being removed, which avoids the need for client >> tools (and I'm not talking just about pip here, but also about one-off >> scripts, which is the sort of thing I write a lot) to trawl through >> all of that data. And it does so without the setuptools project having >> to change how it writes its PyPI page (i.e., the project >> long_description). >> Arguably, that's equally a way of avoiding breaking backward compatibility... >> >>> The second could be done without breakage alltogether i think: at one >>> time all external urls are auto-registered as external indexes >>> and they are presented on the simple page with some meta information >>> that does not confuse older pips/easy_installs. Newer pips/easy_installs >>> can then provide nice error messages. Older pips can continue to use >>> the PEP438 options. And easy install can continue to work. >> >> Setuptools has 255 internal links to files hosted on PyPI. And about >> 11,000 other links. (I just checked that 3 times, as I couldn't >> believe it, but it *seems* to be right :-(). Removing duplicates, 337 >> unique links. Are you suggesting pip presents all of those as possible >> external indexes? > > No, i effectively suggest that PyPI would present just 2 index links, > those which currently are attributed as rel={download,homepage}. > Those two index links would be put into the new "extra indexes field" > on pypi with a note like "the following indexes were extracted from old > release data" which newer pip versions could present to the user. > For older pip/easy_installs things would just continue to work > but they'd get a shorter setuptools simple page. I am not opposed to moving the rel={download,homepage} automatically from link to metatag, I am opposed to leaving them in place in an attempt to prioritize backwards compatibility over safety. The only thing I have against automatically translating the old links to the new external hosting metadata is that it?s going to be a lot of noise for authors who won?t know which links are the correct links to use. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From cody.piersall at gmail.com Tue Oct 7 19:28:25 2014 From: cody.piersall at gmail.com (Cody Piersall) Date: Tue, 7 Oct 2014 12:28:25 -0500 Subject: [Distutils] setup.py develop issues Message-ID: I am having issues with using setup.py develop on my computer at work. My understanding of the command is that it should make my project import-able without really installing it, so that I can continue to change my files without needing to reinstall the project. Here is a minimum example of the problem. I create a new project (named project) with only a single package, and create the minimum setup.py file that I thought would work. mkdir project cd project mkdir project echo. > project\__init__.py echo.from setuptools import setup>setup.py echo.setup(>>setup.py echo. name='project',>>setup.py echo. packages=['project'],>>setup.py echo.^)>>setup.py cd .. After running these commands (as a batch script), I did cd project python setup.py develop but the project was not importable. I can import it when I run `python setup.py install` though. Here is the output of running python setup.py develop: C:\Users\CWP\Development\project>python setup.py develop running develop running egg_info creating project.egg-info writing top-level names to project.egg-info\top_level.txt writing project.egg-info\PKG-INFO writing dependency_links to project.egg-info\dependency_links.txt writing manifest file 'project.egg-info\SOURCES.txt' reading manifest file 'project.egg-info\SOURCES.txt' writing manifest file 'project.egg-info\SOURCES.txt' running build_ext Creating c:\python34\lib\site-packages\project.egg-link (link to .) Adding project 0.0.0 to easy-install.pth file Installed c:\users\cwp\development\project Processing dependencies for project==0.0.0 Finished processing dependencies for project==0.0.0 I'm using Python 3.4.0 and setuptools 6.0.2. I'm not sure if I'm using setuptools wrong, if my understanding of setuptools is wrong, or if this is a bug in setuptools under Windows. Cody -------------- next part -------------- An HTML attachment was scrubbed... URL: From maphew at gmail.com Tue Oct 7 21:29:59 2014 From: maphew at gmail.com (Matt Wilkie) Date: Tue, 7 Oct 2014 12:29:59 -0700 Subject: [Distutils] setup.py develop issues In-Reply-To: References: Message-ID: using pip on your setup.py works for me (in py2.7 though) D:\test> make-project.bat D:\test> pip install --editable project\. Obtaining file:///d%7C%5Ctest%5Cproject Running setup.py egg_info for package from file:///d%7C%5Ctest%5Cproject Installing collected packages: project Running setup.py develop for project Creating c:\python27\arcgis10.2\lib\site-packages\project.egg-link (link to .) Adding project 0.0.0 to easy-install.pth file Installed d:\test\project Successfully installed project Cleaning up... D:\test> python -c "import project" (empty line means success) cheers, -matt On Tue, Oct 7, 2014 at 10:28 AM, Cody Piersall wrote: > I am having issues with using setup.py develop on my computer at work. My > understanding of the command is that it should make my project import-able > without really installing it, so that I can continue to change my files > without needing to reinstall the project. > > Here is a minimum example of the problem. I create a new project (named > project) with only a single package, and create the minimum setup.py file > that I thought would work. > > mkdir project > cd project > mkdir project > echo. > project\__init__.py > echo.from setuptools import setup>setup.py > echo.setup(>>setup.py > echo. name='project',>>setup.py > echo. packages=['project'],>>setup.py > echo.^)>>setup.py > cd .. > > After running these commands (as a batch script), I did > > cd project > python setup.py develop > > but the project was not importable. I can import it when I run `python > setup.py install` though. Here is the output of running python setup.py > develop: > > C:\Users\CWP\Development\project>python setup.py develop > running develop > running egg_info > creating project.egg-info > writing top-level names to project.egg-info\top_level.txt > writing project.egg-info\PKG-INFO > writing dependency_links to project.egg-info\dependency_links.txt > writing manifest file 'project.egg-info\SOURCES.txt' > reading manifest file 'project.egg-info\SOURCES.txt' > writing manifest file 'project.egg-info\SOURCES.txt' > running build_ext > Creating c:\python34\lib\site-packages\project.egg-link (link to .) > Adding project 0.0.0 to easy-install.pth file > Installed c:\users\cwp\development\project > Processing dependencies for project==0.0.0 > Finished processing dependencies for project==0.0.0 > > I'm using Python 3.4.0 and setuptools 6.0.2. I'm not sure if I'm using > setuptools wrong, if my understanding of setuptools is wrong, or if this is > a bug in setuptools under Windows. > > Cody > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cody.piersall at gmail.com Tue Oct 7 23:23:02 2014 From: cody.piersall at gmail.com (Cody Piersall) Date: Tue, 7 Oct 2014 16:23:02 -0500 Subject: [Distutils] setup.py develop issues In-Reply-To: References: Message-ID: On Tue, Oct 7, 2014 at 2:29 PM, Matt Wilkie wrote: > > using pip on your setup.py works for me (in py2.7 though) > > D:\test> make-project.bat > > D:\test> pip install --editable project\. > Obtaining file:///d%7C%5Ctest%5Cproject > Running setup.py egg_info for package from file:///d%7C%5Ctest%5Cproject > > Installing collected packages: project > Running setup.py develop for project > > Creating c:\python27\arcgis10.2\lib\site-packages\project.egg-link (link to .) > Adding project 0.0.0 to easy-install.pth file > > Installed d:\test\project > Successfully installed project > Cleaning up... > > D:\test> python -c "import project" > > (empty line means success) > > cheers, > > -matt Unfortunately, using pip with Python 3 still didn't work for me. Cody -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard at python.org Wed Oct 8 02:23:44 2014 From: richard at python.org (Richard Jones) Date: Wed, 8 Oct 2014 11:23:44 +1100 Subject: [Distutils] Create formal process for claiming 'abandoned' packages In-Reply-To: References: <541CBC32.2000404@stoneleaf.us> <88CAA918-E4D2-428C-ACA1-55219DF6AB0E@stufft.io> Message-ID: Hi folks, sorry for the delay, I was on vacation and then catching up on stuff. I've composed a draft of the policy and I welcome your comments (in the doc, please): https://docs.google.com/document/d/1elum7ENjQb0dLB4ATfYNtnXYVLUzsKacc0VWnHHJb2A/edit?usp=sharing My apologies if I've missed some nuance in a particular contribution to this discussion: please just leave a comment in the doc :) Richard On 23 September 2014 07:49, Nick Coghlan wrote: > > On 23 Sep 2014 00:19, "Antoine Pitrou" wrote: > > > > Donald Stufft stufft.io> writes: > > > > > > PyPI inherinently has complete control over who owns what name on PyPI. > > > > Political authority does not derive from technical control, though. > > > > > As Toshio said that are situations where it makes *obvious* sense to > transfer > > > ownership of a project. Using Django as an pretty good example here, > There are > > > four people able to make releases there, until fairly recently there > were only > > > two if I recall. I don't think anyone would be against PyPI transfering > > > ownership of Django to another active core developer of Django in the > event > > > that all of the people with permissions on PyPI were gone in some > fashion. > > > > Assuming the remaining Django core developers agree on it, then, yes, > that > > can make sense. That's because they are the primary authors of the > project > > (even though they might not have been listed as such on PyPI). > > > > The case people are worried about is whether someone who is not part of > the > > original project author(s) or maintainer(s) can get assigned the PyPI > project. > > In that case people should use one of the forks; there's no reason for > PyPI > > to crown a successor. > > That's why I consider it important to get the original project's issue > tracker involved in the transfer process. I'd also be OK with a process > that required an affirmative "Yes" from the project community, defaulting > to "No transfer" in the case of a lack of response. > > Transfers are most needed for highly active projects where a fork could > have a lot of ripple effects. I think it's reasonable to interpret "nobody > cared enough to say yes or no" as "nobody cares enough for a transfer to be > needed - just fork it rather than claiming the name". > > Regards, > Nick. > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Wed Oct 8 09:17:44 2014 From: holger at merlinux.eu (holger krekel) Date: Wed, 8 Oct 2014 07:17:44 +0000 Subject: [Distutils] PEP470 installation security problems In-Reply-To: <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> References: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> <20141003080629.GG7954@merlinux.eu> <5CD183C2-06B5-44B3-9104-566C7AF0BC76@stufft.io> <20141003182800.GH7954@merlinux.eu> <20141007100925.GM7954@merlinux.eu> <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> Message-ID: <20141008071743.GO7954@merlinux.eu> On Tue, Oct 07, 2014 at 08:00 -0400, Donald Stufft wrote: > > On Oct 7, 2014, at 6:09 AM, holger krekel wrote: > >> I had thought of similar things, and my reasons for not using an >> href> and instead using a meta tag and for removing the old URLs > >> instead of just making this in addition to is: > >> > >> 1. I don?t *want* users of older versions of pip/easy_install to > >> implicitly be fetching these things, they should be able to opt in as > >> well and indeed all the mechanisms exist in pip/easy_install for them > >> to already do so. The only thing that doesn?t exist is the discovery > >> mechanism. > > > > I think it's better to generally avoid deliberately breaking things. > > Things break enough even when we don't intend them to. > > > > IOW, Pypi should IMO aim to preserve working with as many client side > > scenarios as possible -- while adding things and improving for newer > > versions of clients. > > And here I think is where the crux of our disagreement lies I think. > > I think that PyPI should preserve working with as many client side > scenarios as possible, except where there is good reason to do so. > > I believe the fact that the vast bulk of the cases we?d be breaking are > people who are silently, and often unknowingly, being directed to > download some code over unauthenticated channels is a very good reason > to break those cases. Especially given the fact that there is a fairly > trivial work around for people who want to restore that behavior. > > In a way this is similar to switching Python to enforcing TLS verification > by default, which afaik Guido has blessed even for 2.7 assuming that there > is a sane way to restore the default behavior and configure it. Are you saying that PEP470's breaking of backard compatibility is deliberate and helps to defend against MITM attacks during installation? That might be true although i note that hacked servers (see also: bash, ssl) are much more common than MITM attacks and a hacked server can do SSL just fine. In any case, I see two security related downsides of PEP470, one of them severe. For one, current multi-index operations are riskier than PEP438's validated external release file urls. Because currently you only need to trust pypi.python.org has not been hacked but with PEP470 you need to trust the integrity of the external site as well. IIUC you and Nick think this is acceptable because people deliberately make that choice by supplying an explicit option to use the external index, right? If so i think the PEP should also be clear on the fact that Pip/pypi's external repo support is far inferior to typical linux repos because release files are not signed etc. Worse security problems loom with current multi-index ops like the --extra-index-url option which is advertised prominently in PEP470. You recommend to use it for private package indexes, but it can trivially compromise user machines: you register a private package name publically to pypi and add some malware release files, and can then infect all machines which execute an innocent "pip install --extra-index-url ...". I think we conversed about this issue earlier but i don't see the PEP discussing it but rather it recommends using it without a direct call for caution (*). I maintain this attack is more serious than MITM attacks for which you are even ready to break backward compat. Donald, Nick, i am not against the goals of PEP470 per se but in its current form i see it rather causing damage. When i explained to companies the dangers of pip multi-index operations they were rather alarmed and urged me to do something about it within the devpi context. But PEP470 pretends all is fine and everybody should move to multi-index immediately -- that's premature at least if not outright endagering users even today because they take the advise in the draft PEP470 for granted because it comes from Nick and Donald who usually know what they are talking about. At the very least we need to have clear discussion in the PEP about it and safer options for pip and PEP470 needs to MANDATE it for pip and maybe even for easy_install -- you could follow the devpi "pypi_whitelist" design to prevent mixed private/public package links and introduce a "--private-index-url" which means that pip would look first there and when it finds links for a name it would not consider other/public indexes unless the name is explicitely whitelisted. I admit i am not happy about the usability of that but it gives a good secure default against public packages infecting private package installs. best, holger (*) I saw that PEP470 in a different section says "Installers SHOULD implement some mechanism for removing or otherwise disabling use of the default repository." but that's just a "SHOULD" and even if implemented it will not fix fix things retro-actively for older pip/easy_install users -- but you claim fixing things for them is within the PEP470 scope above. From donald at stufft.io Wed Oct 8 09:47:58 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 8 Oct 2014 03:47:58 -0400 Subject: [Distutils] PEP470 installation security problems In-Reply-To: <20141008071743.GO7954@merlinux.eu> References: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> <20141003080629.GG7954@merlinux.eu> <5CD183C2-06B5-44B3-9104-566C7AF0BC76@stufft.io> <20141003182800.GH7954@merlinux.eu> <20141007100925.GM7954@merlinux.eu> <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> Message-ID: <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> > On Oct 8, 2014, at 3:17 AM, holger krekel wrote: > > On Tue, Oct 07, 2014 at 08:00 -0400, Donald Stufft wrote: >>> On Oct 7, 2014, at 6:09 AM, holger krekel wrote: >>>> I had thought of similar things, and my reasons for not using an >>> href> and instead using a meta tag and for removing the old URLs >>>> instead of just making this in addition to is: >>>> >>>> 1. I don?t *want* users of older versions of pip/easy_install to >>>> implicitly be fetching these things, they should be able to opt in as >>>> well and indeed all the mechanisms exist in pip/easy_install for them >>>> to already do so. The only thing that doesn?t exist is the discovery >>>> mechanism. >>> >>> I think it's better to generally avoid deliberately breaking things. >>> Things break enough even when we don't intend them to. >>> >>> IOW, Pypi should IMO aim to preserve working with as many client side >>> scenarios as possible -- while adding things and improving for newer >>> versions of clients. >> >> And here I think is where the crux of our disagreement lies I think. >> >> I think that PyPI should preserve working with as many client side >> scenarios as possible, except where there is good reason to do so. >> >> I believe the fact that the vast bulk of the cases we?d be breaking are >> people who are silently, and often unknowingly, being directed to >> download some code over unauthenticated channels is a very good reason >> to break those cases. Especially given the fact that there is a fairly >> trivial work around for people who want to restore that behavior. >> >> In a way this is similar to switching Python to enforcing TLS verification >> by default, which afaik Guido has blessed even for 2.7 assuming that there >> is a sane way to restore the default behavior and configure it. > > Are you saying that PEP470's breaking of backard compatibility is > deliberate and helps to defend against MITM attacks during installation? > That might be true although i note that hacked servers (see also: bash, > ssl) are much more common than MITM attacks and a hacked server can do > SSL just fine. Yes. > > In any case, I see two security related downsides of PEP470, one of > them severe. > > For one, current multi-index operations are riskier than PEP438's > validated external release file urls. Because currently you only need > to trust pypi.python.org has not been hacked but with PEP470 you > need to trust the integrity of the external site as well. > IIUC you and Nick think this is acceptable because people deliberately > make that choice by supplying an explicit option to use the external > index, right? If so i think the PEP should also be clear on the fact that > Pip/pypi's external repo support is far inferior to typical linux repos > because release files are not signed etc. I might agree with you, except an important consideration of a security feature is, ?Does anybody even use this??. Looking at adoption rates it?s clear that practically nobody *does* use it. If it?s the most secure thing in the world but 95+% of the traffic is using the insecure option, does it really even matter if it?s secure? To be honest, it?s *not* inferior to typical linux repos because in both cases there is an online key you can compromise. If you compromise the debian build fleet you can sign any release files you want, just like if you compromise the Fastly servers and get the PyPI TLS key. You generally do *not* get end to end verification on any Linux repo. The big benefit of the linux model is that it enables untrusted mirrors whereas our current model does not. > > Worse security problems loom with current multi-index ops like > the --extra-index-url option which is advertised prominently in PEP470. > You recommend to use it for private package indexes, but it can > trivially compromise user machines: you register a private package name > publically to pypi and add some malware release files, and can then > infect all machines which execute an innocent "pip install > --extra-index-url ...". I think we conversed about this issue earlier but i > don't see the PEP discussing it but rather it recommends using it > without a direct call for caution (*). I maintain this attack is more > serious than MITM attacks for which you are even ready to break backward > compat. In the context of PEP 470 it?s giving another way for someone who has registered a project on PyPI to host off of PyPI. In this sense there is zero ability for someone else to come along and ?override? the package name. The ability to do this for private projects is really only relevant in that by reusing that mechanism we have a single concept that users need to learn instead of multiple concepts. ?There should be one way to do it?. > > Donald, Nick, i am not against the goals of PEP470 per se but in its > current form i see it rather causing damage. When i explained to companies > the dangers of pip multi-index operations they were rather alarmed and urged > me to do something about it within the devpi context. But PEP470 pretends > all is fine and everybody should move to multi-index immediately -- that's > premature at least if not outright endagering users even today because > they take the advise in the draft PEP470 for granted because it comes > from Nick and Donald who usually know what they are talking about. This is really FUDish. Multi repository support *is* fine. If you have a private project then you should likely claim the name on PyPI because even without multi repository support all it would take is someone running pip on their machine and forgetting to switch to your internal index to attack you too. Can there be more improvements? Absolutely. However this particular problem is an inherent issue with a central repository that anyone can upload too. There are things we can do to make it less of a problem but it?s impossible to ever completely solve it. > > At the very least we need to have clear discussion in the PEP about it > and safer options for pip and PEP470 needs to MANDATE it for pip and > maybe even for easy_install -- you could follow the devpi > "pypi_whitelist" design to prevent mixed private/public package links > and introduce a "--private-index-url" which means that pip would look > first there and when it finds links for a name it would not consider > other/public indexes unless the name is explicitely whitelisted. I > admit i am not happy about the usability of that but it gives a good > secure default against public packages infecting private package installs. > > best, > holger > > (*) I saw that PEP470 in a different section says "Installers SHOULD > implement some mechanism for removing or otherwise disabling use of the > default repository." but that's just a "SHOULD" and even if implemented > it will not fix fix things retro-actively for older pip/easy_install > users -- but you claim fixing things for them is within the PEP470 scope > above. > A PEP can?t really mandate anything to an installer and with PEP 438 I think we found that mandating how things are implemented from on top easily ends up being something that turns out worse in the long run. Pip has no means to improve upon the UX of PEP 438 except by deciding we?re not going to follow the PEP. We?d (I?d?) rather not just throw out what the PEPs say so we generally want to follow things. I have plans (and even a branch!) started to further enhance the multiple repository support in pip. A lot of that is modeled after what yum and apt-get has as far as options go. I am completely and unequivocally against things which mandate much at all to what UX pip presents for these things because I think we can better serve our users by being able to make our own UX decisions. After my experiences with a mandated UX from a PEP I?m at the point where personally I?ll ignore any such mandate in the future where I think there is a better option for pip. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From holger at merlinux.eu Wed Oct 8 10:44:56 2014 From: holger at merlinux.eu (holger krekel) Date: Wed, 8 Oct 2014 08:44:56 +0000 Subject: [Distutils] PEP470 installation security problems In-Reply-To: <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> References: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> <20141003080629.GG7954@merlinux.eu> <5CD183C2-06B5-44B3-9104-566C7AF0BC76@stufft.io> <20141003182800.GH7954@merlinux.eu> <20141007100925.GM7954@merlinux.eu> <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> Message-ID: <20141008084456.GP7954@merlinux.eu> On Wed, Oct 08, 2014 at 03:47 -0400, Donald Stufft wrote: > > On Oct 8, 2014, at 3:17 AM, holger krekel wrote: > > Worse security problems loom with current multi-index ops like > > the --extra-index-url option which is advertised prominently in PEP470. > > You recommend to use it for private package indexes, but it can > > trivially compromise user machines: you register a private package name > > publically to pypi and add some malware release files, and can then > > infect all machines which execute an innocent "pip install > > --extra-index-url ...". I think we conversed about this issue earlier but i > > don't see the PEP discussing it but rather it recommends using it > > without a direct call for caution (*). I maintain this attack is more > > serious than MITM attacks for which you are even ready to break backward > > compat. > > In the context of PEP 470 it?s giving another way for someone who has > registered a project on PyPI to host off of PyPI. In this sense there is > zero ability for someone else to come along and ?override? the package name. > The ability to do this for private projects is really only relevant in that > by reusing that mechanism we have a single concept that users need to learn > instead of multiple concepts. ?There should be one way to do it?. > > > > > Donald, Nick, i am not against the goals of PEP470 per se but in its > > current form i see it rather causing damage. When i explained to companies > > the dangers of pip multi-index operations they were rather alarmed and urged > > me to do something about it within the devpi context. But PEP470 pretends > > all is fine and everybody should move to multi-index immediately -- that's > > premature at least if not outright endagering users even today because > > they take the advise in the draft PEP470 for granted because it comes > > from Nick and Donald who usually know what they are talking about. > > This is really FUDish. Multi repository support *is* fine. If you have a private > project then you should likely claim the name on PyPI because even without > multi repository support all it would take is someone running pip on their > machine and forgetting to switch to your internal index to attack you too. I am sorry if raising the issue of private/public compromises sounds like FUD to you. From my experience it's a real attack vector. I talked about this at EP2014 (http://youtu.be/aNrrGf-uNUY?t=6m1s ) and people got back to me afterwards, surprised. And I don't think you can successfully ask people in companies around the work to register private package names publically (let alone the issue of clashes etc.). Admit it, that's even more unlikely than peple using some PEP438 features :) And yes, if someone forgets to set the private index he could still pull in malicious public links even with devpi or new pip options. > Can there be more improvements? Absolutely. However this particular problem > is an inherent issue with a central repository that anyone can upload too. > There are things we can do to make it less of a problem but it?s impossible > to ever completely solve it. Linux repos are totally different: their main index is a curated index and pypi's is a wiki. Thus merging links from a private index and the pypi wiki can trivially wreak havoc while putting malware into the central Debian or Redhat repo is very hard. > > At the very least we need to have clear discussion in the PEP about it > > and safer options for pip and PEP470 needs to MANDATE it for pip and > > maybe even for easy_install -- you could follow the devpi > > "pypi_whitelist" design to prevent mixed private/public package links > > and introduce a "--private-index-url" which means that pip would look > > first there and when it finds links for a name it would not consider > > other/public indexes unless the name is explicitely whitelisted. I > > admit i am not happy about the usability of that but it gives a good > > secure default against public packages infecting private package installs. > > > > best, > > holger > > > > (*) I saw that PEP470 in a different section says "Installers SHOULD > > implement some mechanism for removing or otherwise disabling use of the > > default repository." but that's just a "SHOULD" and even if implemented > > it will not fix fix things retro-actively for older pip/easy_install > > users -- but you claim fixing things for them is within the PEP470 scope > > above. > > > > A PEP can?t really mandate anything to an installer and with PEP 438 I > think we found that mandating how things are implemented from on top easily > ends up being something that turns out worse in the long run. UI design is a delicate thing -- but i am sure you remember that you were involved in PEP438 and actually pushed for some UI that you are now criticising. I am a bit irritated but i understand that you probably all along wanted to push the processes towards the "multi-repo" idea. Please note that i am not against this in principle. > Pip has no means to improve upon the UX of PEP 438 except by deciding > we?re not going to follow the PEP. We?d (I?d?) rather not just throw > out what the PEPs say so we generally want to follow things. And that's a good thing, thanks! Given the importance of PyPI today in the python community, I think the way how PyPI interacts with tools and installers deserves PEPs. > I have plans (and even a branch!) started to further enhance the multiple > repository support in pip. A lot of that is modeled after what yum and apt-get > has as far as options go. I am completely and unequivocally against things > which mandate much at all to what UX pip presents for these things because > I think we can better serve our users by being able to make our own UX decisions. > After my experiences with a mandated UX from a PEP I?m at the point where > personally I?ll ignore any such mandate in the future where I think there > is a better option for pip. PEPs are a form of helping collaboration and growth in a community but certainly not the only way and, if done badly, can do more damage than good. best, holger From donald at stufft.io Wed Oct 8 11:44:36 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 8 Oct 2014 05:44:36 -0400 Subject: [Distutils] PEP470 installation security problems In-Reply-To: <20141008084456.GP7954@merlinux.eu> References: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> <20141003080629.GG7954@merlinux.eu> <5CD183C2-06B5-44B3-9104-566C7AF0BC76@stufft.io> <20141003182800.GH7954@merlinux.eu> <20141007100925.GM7954@merlinux.eu> <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> Message-ID: > On Oct 8, 2014, at 4:44 AM, holger krekel wrote: > > On Wed, Oct 08, 2014 at 03:47 -0400, Donald Stufft wrote: >>> On Oct 8, 2014, at 3:17 AM, holger krekel wrote: >>> Worse security problems loom with current multi-index ops like >>> the --extra-index-url option which is advertised prominently in PEP470. >>> You recommend to use it for private package indexes, but it can >>> trivially compromise user machines: you register a private package name >>> publically to pypi and add some malware release files, and can then >>> infect all machines which execute an innocent "pip install >>> --extra-index-url ...". I think we conversed about this issue earlier but i >>> don't see the PEP discussing it but rather it recommends using it >>> without a direct call for caution (*). I maintain this attack is more >>> serious than MITM attacks for which you are even ready to break backward >>> compat. >> >> In the context of PEP 470 it?s giving another way for someone who has >> registered a project on PyPI to host off of PyPI. In this sense there is >> zero ability for someone else to come along and ?override? the package name. >> The ability to do this for private projects is really only relevant in that >> by reusing that mechanism we have a single concept that users need to learn >> instead of multiple concepts. ?There should be one way to do it?. >> >>> >>> Donald, Nick, i am not against the goals of PEP470 per se but in its >>> current form i see it rather causing damage. When i explained to companies >>> the dangers of pip multi-index operations they were rather alarmed and urged >>> me to do something about it within the devpi context. But PEP470 pretends >>> all is fine and everybody should move to multi-index immediately -- that's >>> premature at least if not outright endagering users even today because >>> they take the advise in the draft PEP470 for granted because it comes >>> from Nick and Donald who usually know what they are talking about. >> >> This is really FUDish. Multi repository support *is* fine. If you have a private >> project then you should likely claim the name on PyPI because even without >> multi repository support all it would take is someone running pip on their >> machine and forgetting to switch to your internal index to attack you too. > > I am sorry if raising the issue of private/public compromises sounds > like FUD to you. From my experience it's a real attack vector. I talked > about this at EP2014 (http://youtu.be/aNrrGf-uNUY?t=6m1s ) and people > got back to me afterwards, surprised. > > And I don't think you can successfully ask people in companies > around the work to register private package names publically (let alone > the issue of clashes etc.). Admit it, that's even more unlikely than > peple using some PEP438 features :) > > And yes, if someone forgets to set the private index he could still pull > in malicious public links even with devpi or new pip options. I think raising the issue is FUDish because it has nothing to do with using multi repository support for things that are registered on PyPI. The attack vector you?re describing isn?t possible at all for any project that is effected by PEP 470, which are projects which wish to register themselves in the PyPI index without using PyPI as their repository. The *only* reason using the multiple repository support for private hosting is relevant is because it?s the option for allowing people to have private repositories at all, and we can re-use that behavior with this. It?s not particularly relevant to PEP 470 unless you have a suggestion for another, better mechanism that can satisfy all of these use cases (and possibly more?). That being said, the things I have sketched out for pip includes the ability to have both a whitelist and a black list for each repository. I don?t however think that it?s the PEPs place to dictate how that looks (or even if it exists). I?m also not against adding another *SHOULD* saying that installers should implement some mechanism that allows for whitelisting or blacklisting which repository particular projects come from. > >> Can there be more improvements? Absolutely. However this particular problem >> is an inherent issue with a central repository that anyone can upload too. >> There are things we can do to make it less of a problem but it?s impossible >> to ever completely solve it. > > Linux repos are totally different: their main index is a curated index > and pypi's is a wiki. Thus merging links from a private index and > the pypi wiki can trivially wreak havoc while putting malware into > the central Debian or Redhat repo is very hard. > >>> At the very least we need to have clear discussion in the PEP about it >>> and safer options for pip and PEP470 needs to MANDATE it for pip and >>> maybe even for easy_install -- you could follow the devpi >>> "pypi_whitelist" design to prevent mixed private/public package links >>> and introduce a "--private-index-url" which means that pip would look >>> first there and when it finds links for a name it would not consider >>> other/public indexes unless the name is explicitely whitelisted. I >>> admit i am not happy about the usability of that but it gives a good >>> secure default against public packages infecting private package installs. >>> >>> best, >>> holger >>> >>> (*) I saw that PEP470 in a different section says "Installers SHOULD >>> implement some mechanism for removing or otherwise disabling use of the >>> default repository." but that's just a "SHOULD" and even if implemented >>> it will not fix fix things retro-actively for older pip/easy_install >>> users -- but you claim fixing things for them is within the PEP470 scope >>> above. >>> >> >> A PEP can?t really mandate anything to an installer and with PEP 438 I >> think we found that mandating how things are implemented from on top easily >> ends up being something that turns out worse in the long run. > > UI design is a delicate thing -- but i am sure you remember that > you were involved in PEP438 and actually pushed for some UI that you are > now criticising. I am a bit irritated but i understand that you probably > all along wanted to push the processes towards the "multi-repo" idea. > Please note that i am not against this in principle. Absolutely. I don?t think that it was clear at the time that the PEP 438 UX would be as bad as it turned out to be. My take away from that isn?t so much that the people involved were bad at UXs but that codifying a UX into a PEP is a bad idea in general. With PEP 470 you can see this because even on the PyPI side I don?t dictate a UX in it. I spell out the API that I expect PyPI to adopt but I don?t mention what the UX looks like so that we can easily adjust it. I also don?t spell out what the UXs on the installers look like, against because It?s my belief now that dictating UX is a generally bad idea, instead I spell out what features an installer *should* have. This is all just lessons learned from trying to spell out a UX inside of a PEP, we did it, it didn?t work and when it didn?t work the fact it was in a PEP put us in a crappy situation of having to either write a whole new PEP (and possible recreate new UX issues) or start ignoring PEPs. For the record, both pip and easy_install already have mechanisms for disabling the default repository. In pip this is ``?no-index`` and in easy_install it?s not as easy but you can either override it with ``?index-url`` or use the ``?allow-hosts`` option to disallow PyPI. > >> Pip has no means to improve upon the UX of PEP 438 except by deciding >> we?re not going to follow the PEP. We?d (I?d?) rather not just throw >> out what the PEPs say so we generally want to follow things. > > And that's a good thing, thanks! Given the importance of PyPI today in > the python community, I think the way how PyPI interacts with tools and > installers deserves PEPs. Yea I agree, which is why I?m trying to figure out how to do PEPs without making them feel more like handcuffs than useful tools :) > >> I have plans (and even a branch!) started to further enhance the multiple >> repository support in pip. A lot of that is modeled after what yum and apt-get >> has as far as options go. I am completely and unequivocally against things >> which mandate much at all to what UX pip presents for these things because >> I think we can better serve our users by being able to make our own UX decisions. >> After my experiences with a mandated UX from a PEP I?m at the point where >> personally I?ll ignore any such mandate in the future where I think there >> is a better option for pip. > > PEPs are a form of helping collaboration and growth in a community but > certainly not the only way and, if done badly, can do more damage than good. > > best, > holger --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From holger at merlinux.eu Wed Oct 8 12:06:59 2014 From: holger at merlinux.eu (holger krekel) Date: Wed, 8 Oct 2014 10:06:59 +0000 Subject: [Distutils] PEP470 installation security problems In-Reply-To: References: <5CD183C2-06B5-44B3-9104-566C7AF0BC76@stufft.io> <20141003182800.GH7954@merlinux.eu> <20141007100925.GM7954@merlinux.eu> <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> Message-ID: <20141008100659.GQ7954@merlinux.eu> On Wed, Oct 08, 2014 at 05:44 -0400, Donald Stufft wrote: > > On Oct 8, 2014, at 4:44 AM, holger krekel wrote: > > > > On Wed, Oct 08, 2014 at 03:47 -0400, Donald Stufft wrote: > >>> On Oct 8, 2014, at 3:17 AM, holger krekel wrote: > >>> Worse security problems loom with current multi-index ops like > >>> the --extra-index-url option which is advertised prominently in PEP470. > >>> You recommend to use it for private package indexes, but it can > >>> trivially compromise user machines: you register a private package name > >>> publically to pypi and add some malware release files, and can then > >>> infect all machines which execute an innocent "pip install > >>> --extra-index-url ...". I think we conversed about this issue earlier but i > >>> don't see the PEP discussing it but rather it recommends using it > >>> without a direct call for caution (*). I maintain this attack is more > >>> serious than MITM attacks for which you are even ready to break backward > >>> compat. > >> > >> In the context of PEP 470 it?s giving another way for someone who has > >> registered a project on PyPI to host off of PyPI. In this sense there is > >> zero ability for someone else to come along and ?override? the package name. > >> The ability to do this for private projects is really only relevant in that > >> by reusing that mechanism we have a single concept that users need to learn > >> instead of multiple concepts. ?There should be one way to do it?. > >> > >>> > >>> Donald, Nick, i am not against the goals of PEP470 per se but in its > >>> current form i see it rather causing damage. When i explained to companies > >>> the dangers of pip multi-index operations they were rather alarmed and urged > >>> me to do something about it within the devpi context. But PEP470 pretends > >>> all is fine and everybody should move to multi-index immediately -- that's > >>> premature at least if not outright endagering users even today because > >>> they take the advise in the draft PEP470 for granted because it comes > >>> from Nick and Donald who usually know what they are talking about. > >> > >> This is really FUDish. Multi repository support *is* fine. If you have a private > >> project then you should likely claim the name on PyPI because even without > >> multi repository support all it would take is someone running pip on their > >> machine and forgetting to switch to your internal index to attack you too. > > > > I am sorry if raising the issue of private/public compromises sounds > > like FUD to you. From my experience it's a real attack vector. I talked > > about this at EP2014 (http://youtu.be/aNrrGf-uNUY?t=6m1s ) and people > > got back to me afterwards, surprised. > > > > And I don't think you can successfully ask people in companies > > around the work to register private package names publically (let alone > > the issue of clashes etc.). Admit it, that's even more unlikely than > > peple using some PEP438 features :) > > > > And yes, if someone forgets to set the private index he could still pull > > in malicious public links even with devpi or new pip options. > > I think raising the issue is FUDish because it has nothing to do with using > multi repository support for things that are registered on PyPI. Well, the PEP has two central paragraphs motivating multi-index operations: The two common installer tools, pip and easy_install/setuptools, both support the concept of additional locations to search for files to satisify the installation requirements and have done so for many years. This means that there is no need to "phase" in a new flag or concept and the solution to installing a project from a repository other than PyPI will function regardless of how old (within reason) the end user's installer is. Not only has this concept existed in the Python tooling for some time, but it is a concept that exists across languages and even extending to the OS level with OS package tools almost universally using multiple repository support making it extremely likely that someone is already familar with the concept. Additionally, the multiple repository approach is a concept that is useful outside of the narrow scope of allowing projects which wish to be included on the index portion of PyPI but do not wish to utilize the repository portion of PyPI. This includes places where a company may wish to host a repository that contains their internal packages or where a project may wish to have multiple "channels" of releases, such as alpha, beta, release candidate, and final release. and then it concretely suggests "--extra-index-url" and gives an example. It does not say that this is only good if you are using private projects that have a presence on PyPI. It rather suggests multi-index is the thing to go for today, generally, does it not? Given that PyPI is a wiki and Linux Distros are a curated index, i insist it's dangerous to recommend to mix multiple indexes with pip if you don't know quite exactly what you are doing. Do you really disagree on this? best, holger > The attack > vector you?re describing isn?t possible at all for any project that is effected > by PEP 470, which are projects which wish to register themselves in the PyPI > index without using PyPI as their repository. > > The *only* reason using the multiple repository support for private hosting is > relevant is because it?s the option for allowing people to have private > repositories at all, and we can re-use that behavior with this. It?s not particularly > relevant to PEP 470 unless you have a suggestion for another, better mechanism > that can satisfy all of these use cases (and possibly more?). > > That being said, the things I have sketched out for pip includes the ability to > have both a whitelist and a black list for each repository. I don?t however think > that it?s the PEPs place to dictate how that looks (or even if it exists). > > I?m also not against adding another *SHOULD* saying that installers should > implement some mechanism that allows for whitelisting or blacklisting which > repository particular projects come from. > > > > >> Can there be more improvements? Absolutely. However this particular problem > >> is an inherent issue with a central repository that anyone can upload too. > >> There are things we can do to make it less of a problem but it?s impossible > >> to ever completely solve it. > > > > Linux repos are totally different: their main index is a curated index > > and pypi's is a wiki. Thus merging links from a private index and > > the pypi wiki can trivially wreak havoc while putting malware into > > the central Debian or Redhat repo is very hard. > > > >>> At the very least we need to have clear discussion in the PEP about it > >>> and safer options for pip and PEP470 needs to MANDATE it for pip and > >>> maybe even for easy_install -- you could follow the devpi > >>> "pypi_whitelist" design to prevent mixed private/public package links > >>> and introduce a "--private-index-url" which means that pip would look > >>> first there and when it finds links for a name it would not consider > >>> other/public indexes unless the name is explicitely whitelisted. I > >>> admit i am not happy about the usability of that but it gives a good > >>> secure default against public packages infecting private package installs. > >>> > >>> best, > >>> holger > >>> > >>> (*) I saw that PEP470 in a different section says "Installers SHOULD > >>> implement some mechanism for removing or otherwise disabling use of the > >>> default repository." but that's just a "SHOULD" and even if implemented > >>> it will not fix fix things retro-actively for older pip/easy_install > >>> users -- but you claim fixing things for them is within the PEP470 scope > >>> above. > >>> > >> > >> A PEP can?t really mandate anything to an installer and with PEP 438 I > >> think we found that mandating how things are implemented from on top easily > >> ends up being something that turns out worse in the long run. > > > > UI design is a delicate thing -- but i am sure you remember that > > you were involved in PEP438 and actually pushed for some UI that you are > > now criticising. I am a bit irritated but i understand that you probably > > all along wanted to push the processes towards the "multi-repo" idea. > > Please note that i am not against this in principle. > > Absolutely. I don?t think that it was clear at the time that the PEP 438 UX > would be as bad as it turned out to be. My take away from that isn?t so much > that the people involved were bad at UXs but that codifying a UX into a PEP > is a bad idea in general. With PEP 470 you can see this because even on the > PyPI side I don?t dictate a UX in it. I spell out the API that I expect > PyPI to adopt but I don?t mention what the UX looks like so that we can > easily adjust it. I also don?t spell out what the UXs on the installers look > like, against because It?s my belief now that dictating UX is a generally bad > idea, instead I spell out what features an installer *should* have. > > This is all just lessons learned from trying to spell out a UX inside of a PEP, > we did it, it didn?t work and when it didn?t work the fact it was in a PEP put > us in a crappy situation of having to either write a whole new PEP (and possible > recreate new UX issues) or start ignoring PEPs. > > For the record, both pip and easy_install already have mechanisms for disabling > the default repository. In pip this is ``?no-index`` and in easy_install it?s > not as easy but you can either override it with ``?index-url`` or use the > ``?allow-hosts`` option to disallow PyPI. > > > > >> Pip has no means to improve upon the UX of PEP 438 except by deciding > >> we?re not going to follow the PEP. We?d (I?d?) rather not just throw > >> out what the PEPs say so we generally want to follow things. > > > > And that's a good thing, thanks! Given the importance of PyPI today in > > the python community, I think the way how PyPI interacts with tools and > > installers deserves PEPs. > > Yea I agree, which is why I?m trying to figure out how to do PEPs without > making them feel more like handcuffs than useful tools :) > > > > >> I have plans (and even a branch!) started to further enhance the multiple > >> repository support in pip. A lot of that is modeled after what yum and apt-get > >> has as far as options go. I am completely and unequivocally against things > >> which mandate much at all to what UX pip presents for these things because > >> I think we can better serve our users by being able to make our own UX decisions. > >> After my experiences with a mandated UX from a PEP I?m at the point where > >> personally I?ll ignore any such mandate in the future where I think there > >> is a better option for pip. > > > > PEPs are a form of helping collaboration and growth in a community but > > certainly not the only way and, if done badly, can do more damage than good. > > > > best, > > holger > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > From donald at stufft.io Wed Oct 8 12:24:02 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 8 Oct 2014 06:24:02 -0400 Subject: [Distutils] PEP470 installation security problems In-Reply-To: <20141008100659.GQ7954@merlinux.eu> References: <5CD183C2-06B5-44B3-9104-566C7AF0BC76@stufft.io> <20141003182800.GH7954@merlinux.eu> <20141007100925.GM7954@merlinux.eu> <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008100659.GQ7954@merlinux.eu> Message-ID: > On Oct 8, 2014, at 6:06 AM, holger krekel wrote: > > On Wed, Oct 08, 2014 at 05:44 -0400, Donald Stufft wrote: >> >> I think raising the issue is FUDish because it has nothing to do with using >> multi repository support for things that are registered on PyPI. > > Well, the PEP has two central paragraphs motivating multi-index operations: > > The two common installer tools, pip and easy_install/setuptools, both > support the concept of additional locations to search for files to > satisify the installation requirements and have done so for many years. > This means that there is no need to "phase" in a new flag or concept and > the solution to installing a project from a repository other than PyPI > will function regardless of how old (within reason) the end user's > installer is. Not only has this concept existed in the Python tooling > for some time, but it is a concept that exists across languages and even > extending to the OS level with OS package tools almost universally using > multiple repository support making it extremely likely that someone is > already familar with the concept. > > Additionally, the multiple repository approach is a concept that is > useful outside of the narrow scope of allowing projects which wish > to be included on the index portion of PyPI but do not wish to > utilize the repository portion of PyPI. This includes places where a > company may wish to host a repository that contains their internal > packages or where a project may wish to have multiple "channels" of > releases, such as alpha, beta, release candidate, and final release. > > and then it concretely suggests "--extra-index-url" and gives an example. > It does not say that this is only good if you are using private projects > that have a presence on PyPI. It rather suggests multi-index is the thing > to go for today, generally, does it not? > > Given that PyPI is a wiki and Linux Distros are a curated index, i > insist it's dangerous to recommend to mix multiple indexes with pip if > you don't know quite exactly what you are doing. Do you really disagree > on this? It is not dangerous to mix multiple indexes in the case that PEP 470 is specifying, which is when you want to have files for a project listed on the PyPI index hosted on a different repository. The use of --extra-index-url in PEP 470 is to show how someone would add one of the extra repositories for a project that is indexed on PyPI, which is again roughly as safe as installing from PyPI at all. If you use the multiple repository support to install things which are not claimed on PyPI and you do not disable the PyPI index, then yes that is dangerous. It also has nothing to do with whether it's safe for someone to add an additional repository that points to the repository that PIL is located at. I've also never suggested to anyone that their company should rely on PyPI and instead I point them towards either making their own repository with Apache/Index/Twisted Web or using devpi. My goal is to make PyPI as safe as possible for people who don't do that, but there are limits to what is possible. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From ncoghlan at gmail.com Wed Oct 8 12:32:33 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 8 Oct 2014 20:32:33 +1000 Subject: [Distutils] PEP470 installation security problems In-Reply-To: <20141008100659.GQ7954@merlinux.eu> References: <5CD183C2-06B5-44B3-9104-566C7AF0BC76@stufft.io> <20141003182800.GH7954@merlinux.eu> <20141007100925.GM7954@merlinux.eu> <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008100659.GQ7954@merlinux.eu> Message-ID: On 8 October 2014 20:06, holger krekel wrote: > Given that PyPI is a wiki and Linux Distros are a curated index, i > insist it's dangerous to recommend to mix multiple indexes with pip if > you don't know quite exactly what you are doing. Do you really disagree > on this? Hence this line in the PEP: End users wishing to limit what files they pull from which repository can simply use devpi to whitelist projects from PyPI or another repository. Anyone running a private PyPI mirror without disabling the use of upstream indexes entirely is already running their infrastructure in a dangerously insecure configuration. That has nothing to do with PEP 470. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From holger at merlinux.eu Wed Oct 8 12:33:00 2014 From: holger at merlinux.eu (holger krekel) Date: Wed, 8 Oct 2014 10:33:00 +0000 Subject: [Distutils] PEP470 installation security problems In-Reply-To: References: <20141003182800.GH7954@merlinux.eu> <20141007100925.GM7954@merlinux.eu> <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008100659.GQ7954@merlinux.eu> Message-ID: <20141008103300.GR7954@merlinux.eu> On Wed, Oct 08, 2014 at 06:24 -0400, Donald Stufft wrote: > > On Oct 8, 2014, at 6:06 AM, holger krekel wrote: > > > > On Wed, Oct 08, 2014 at 05:44 -0400, Donald Stufft wrote: > >> > >> I think raising the issue is FUDish because it has nothing to do with using > >> multi repository support for things that are registered on PyPI. > > > > Well, the PEP has two central paragraphs motivating multi-index operations: > > > > The two common installer tools, pip and easy_install/setuptools, both > > support the concept of additional locations to search for files to > > satisify the installation requirements and have done so for many years. > > This means that there is no need to "phase" in a new flag or concept and > > the solution to installing a project from a repository other than PyPI > > will function regardless of how old (within reason) the end user's > > installer is. Not only has this concept existed in the Python tooling > > for some time, but it is a concept that exists across languages and even > > extending to the OS level with OS package tools almost universally using > > multiple repository support making it extremely likely that someone is > > already familar with the concept. > > > > Additionally, the multiple repository approach is a concept that is > > useful outside of the narrow scope of allowing projects which wish > > to be included on the index portion of PyPI but do not wish to > > utilize the repository portion of PyPI. This includes places where a > > company may wish to host a repository that contains their internal > > packages or where a project may wish to have multiple "channels" of > > releases, such as alpha, beta, release candidate, and final release. > > > > and then it concretely suggests "--extra-index-url" and gives an example. > > It does not say that this is only good if you are using private projects > > that have a presence on PyPI. It rather suggests multi-index is the thing > > to go for today, generally, does it not? > > > > Given that PyPI is a wiki and Linux Distros are a curated index, i > > insist it's dangerous to recommend to mix multiple indexes with pip if > > you don't know quite exactly what you are doing. Do you really disagree > > on this? > > It is not dangerous to mix multiple indexes in the case that PEP 470 is > specifying, which is when you want to have files for a project listed on the > PyPI index hosted on a different repository. Yes, that case is not more dangerous than today. > The use of --extra-index-url in > PEP 470 is to show how someone would add one of the extra repositories for a > project that is indexed on PyPI, which is again roughly as safe as installing > from PyPI at all. Then we are reading the sections i cite above very differently -- IMO you and the PEP generally push for multi-index ops without explaining the risks. Maybe someone else can chime in. best, holger > If you use the multiple repository support to install things which are not > claimed on PyPI and you do not disable the PyPI index, then yes that is > dangerous. It also has nothing to do with whether it's safe for someone to > add an additional repository that points to the repository that PIL is located > at. > > I've also never suggested to anyone that their company should rely on PyPI > and instead I point them towards either making their own repository with > Apache/Index/Twisted Web or using devpi. My goal is to make PyPI as safe as > possible for people who don't do that, but there are limits to what is possible. > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > From ncoghlan at gmail.com Wed Oct 8 12:27:59 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 8 Oct 2014 20:27:59 +1000 Subject: [Distutils] PEP470 installation security problems In-Reply-To: References: <73F13DC5-6E67-4516-874D-8630159B4DAF@stufft.io> <20141003080629.GG7954@merlinux.eu> <5CD183C2-06B5-44B3-9104-566C7AF0BC76@stufft.io> <20141003182800.GH7954@merlinux.eu> <20141007100925.GM7954@merlinux.eu> <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> Message-ID: On 8 October 2014 19:44, Donald Stufft wrote: >> On Oct 8, 2014, at 4:44 AM, holger krekel wrote: >> I am sorry if raising the issue of private/public compromises sounds >> like FUD to you. From my experience it's a real attack vector. I talked >> about this at EP2014 (http://youtu.be/aNrrGf-uNUY?t=6m1s ) and people >> got back to me afterwards, surprised. >> >> And I don't think you can successfully ask people in companies >> around the work to register private package names publically (let alone >> the issue of clashes etc.). Admit it, that's even more unlikely than >> peple using some PEP438 features :) >> >> And yes, if someone forgets to set the private index he could still pull >> in malicious public links even with devpi or new pip options. > > I think raising the issue is FUDish because it has nothing to do with using > multi repository support for things that are registered on PyPI. The attack > vector you?re describing isn?t possible at all for any project that is effected > by PEP 470, which are projects which wish to register themselves in the PyPI > index without using PyPI as their repository. >From my perspective, there's also the question of "relative risk". As soon as anyone is installing anything at all directly from PyPI, their perceived threat level should already be through the roof. Why? Because of .pth files. .pth files run automatically at every interpreter startup, so if you're installing directly from PyPI, then *every single package you install* has the power to completely subvert your application (if you're installing into an application specific virtual environment) or your entire Python installation (if you're installing into a shared Python instance). Even if they don't install malicious .pth files, then many of them are going to get imported at some point anyway, so they're going to able to reach out and do whatever they want to the Python level internal process state. "pip install python-nation" helps illustrate the degree to which we're generally trusting folks uploading stuff to PyPI to not be evil, and that level of trust also extends to folks providing external repositories rather than hosting directly on PyPI. If folks are more worried about the risk of PyPI or a third party repo shadowing their private packages than they are about malicious .pth files or generally malicious runtime behaviour in dependencies, then I strongly believe their threat meters need recalibrating. We focus on MITM attacks in the upstream infrastructure, because if *developers* are actively malicious, then you're already hosed - they don't need to do anything clever, they can just decide to own your system as a side effect of running their code. (Most of them won't, which is why the risk is low in practice. But as far as theoretical attacks go, this is near the top of my personal threat model, just behind third party MITM attacks) If folks are using Python in a context where these risks are unacceptable to them, then they should either be getting their packages via a trusted third party (like a community or commercial Linux distribution, or a commercial Python redistributor), or at least using a PyPI caching proxy with whitelisting support (which is why PEP 470 recommends the use of devpi in conjunction with turning off the default index). Establishing trustworthiness is expensive and relatively slow, which is why PyPI, like a lot of language specific distribution systems, doesn't currently offer it as feature. The generally more limited and older selection of packages in the redistributor channels reflects some of the impact of those additional costs. Bringing those costs down is something we're going to have to fix on the redistributor side - upstream initiatives like PEP 426 may help, but a lot of it is going to be a matter of Linux distros reassessing what services we're able to provide to the wider open source community. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed Oct 8 12:44:19 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 8 Oct 2014 20:44:19 +1000 Subject: [Distutils] PEP470 installation security problems In-Reply-To: <20141008103300.GR7954@merlinux.eu> References: <20141003182800.GH7954@merlinux.eu> <20141007100925.GM7954@merlinux.eu> <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008100659.GQ7954@merlinux.eu> <20141008103300.GR7954@merlinux.eu> Message-ID: On 8 October 2014 20:33, holger krekel wrote: > > Then we are reading the sections i cite above very differently -- IMO > you and the PEP generally push for multi-index ops without explaining > the risks. Note that this explanation is present in the PEP: Currently both pip and setuptools implement multiple repository support by using the best installation candidate it can find from either repository, essentially treating it as if it were one large repository. Is it mainly that you would like the consequences of that in terms of any listed index being able to provide any requested package to be spelled out more clearly? Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From holger at merlinux.eu Wed Oct 8 12:57:36 2014 From: holger at merlinux.eu (holger krekel) Date: Wed, 8 Oct 2014 10:57:36 +0000 Subject: [Distutils] PEP470 installation security problems In-Reply-To: References: <20141003182800.GH7954@merlinux.eu> <20141007100925.GM7954@merlinux.eu> <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> Message-ID: <20141008105736.GS7954@merlinux.eu> On Wed, Oct 08, 2014 at 20:27 +1000, Nick Coghlan wrote: > On 8 October 2014 19:44, Donald Stufft wrote: > >> On Oct 8, 2014, at 4:44 AM, holger krekel wrote: > >> I am sorry if raising the issue of private/public compromises sounds > >> like FUD to you. From my experience it's a real attack vector. I talked > >> about this at EP2014 (http://youtu.be/aNrrGf-uNUY?t=6m1s ) and people > >> got back to me afterwards, surprised. > >> > >> And I don't think you can successfully ask people in companies > >> around the work to register private package names publically (let alone > >> the issue of clashes etc.). Admit it, that's even more unlikely than > >> peple using some PEP438 features :) > >> > >> And yes, if someone forgets to set the private index he could still pull > >> in malicious public links even with devpi or new pip options. > > > > I think raising the issue is FUDish because it has nothing to do with using > > multi repository support for things that are registered on PyPI. The attack > > vector you?re describing isn?t possible at all for any project that is effected > > by PEP 470, which are projects which wish to register themselves in the PyPI > > index without using PyPI as their repository. > > >From my perspective, there's also the question of "relative risk". As > soon as anyone is installing anything at all directly from PyPI, their > perceived threat level should already be through the roof. Why? > Because of .pth files. Well, for installing NAME from pypi you need to trust that the people who registered and maintain NAME are not doing something bad (and the machine is not compromised but in that case all bets are off obviously). And i can make a choice to trust "django", "flask, "warehouse" and other pypi names. I am exposing myself to whatever the maintainers published but it's my choice. This is a very different thing compared to: pip install --extra-index http://private.repo mypackage I may think i am trusting just "mypackage" from my private repo. But in fact i am betting on nobody uploading "mypackage" to the pypi wiki. I don't think this is very obvious to many -- it certainly wasn't at EuroPython2014. best, holger > .pth files run automatically at every interpreter startup, so if > you're installing directly from PyPI, then *every single package you > install* has the power to completely subvert your application (if > you're installing into an application specific virtual environment) or > your entire Python installation (if you're installing into a shared > Python instance). Even if they don't install malicious .pth files, > then many of them are going to get imported at some point anyway, so > they're going to able to reach out and do whatever they want to the > Python level internal process state. > > "pip install python-nation" helps illustrate the degree to which we're > generally trusting folks uploading stuff to PyPI to not be evil, and > that level of trust also extends to folks providing external > repositories rather than hosting directly on PyPI. > > If folks are more worried about the risk of PyPI or a third party repo > shadowing their private packages than they are about malicious .pth > files or generally malicious runtime behaviour in dependencies, then I > strongly believe their threat meters need recalibrating. We focus on > MITM attacks in the upstream infrastructure, because if *developers* > are actively malicious, then you're already hosed - they don't need to > do anything clever, they can just decide to own your system as a side > effect of running their code. (Most of them won't, which is why the > risk is low in practice. But as far as theoretical attacks go, this is > near the top of my personal threat model, just behind third party MITM > attacks) > > If folks are using Python in a context where these risks are > unacceptable to them, then they should either be getting their > packages via a trusted third party (like a community or commercial > Linux distribution, or a commercial Python redistributor), or at least > using a PyPI caching proxy with whitelisting support (which is why PEP > 470 recommends the use of devpi in conjunction with turning off the > default index). > > Establishing trustworthiness is expensive and relatively slow, which > is why PyPI, like a lot of language specific distribution systems, > doesn't currently offer it as feature. The generally more limited and > older selection of packages in the redistributor channels reflects > some of the impact of those additional costs. Bringing those costs > down is something we're going to have to fix on the redistributor side > - upstream initiatives like PEP 426 may help, but a lot of it is going > to be a matter of Linux distros reassessing what services we're able > to provide to the wider open source community. > > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > From p.f.moore at gmail.com Wed Oct 8 13:03:49 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 8 Oct 2014 12:03:49 +0100 Subject: [Distutils] PEP470 installation security problems In-Reply-To: <20141008103300.GR7954@merlinux.eu> References: <20141003182800.GH7954@merlinux.eu> <20141007100925.GM7954@merlinux.eu> <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008100659.GQ7954@merlinux.eu> <20141008103300.GR7954@merlinux.eu> Message-ID: On 8 October 2014 11:33, holger krekel wrote: >> The use of --extra-index-url in >> PEP 470 is to show how someone would add one of the extra repositories for a >> project that is indexed on PyPI, which is again roughly as safe as installing >> from PyPI at all. > > Then we are reading the sections i cite above very differently -- IMO > you and the PEP generally push for multi-index ops without explaining > the risks. > > Maybe someone else can chime in. Chiming in because you asked for other opinions, although I've not yet read to the end of the thread... I read this section, and indeed the whole of the PEP, as basically saying: 1. We have a problem because PEP 438 didn't turn out so well in practice. 2. We have an existing mechanism (multi-index support). 3. The existing mechanism can be used as follows to better solve the problem PEP 438 tried to solve. I don't see any "encouragement" to use multi-index support, other than in the specific case PEP 438 was aimed at. Obviously PEP 470 raises the profile of multi-index support, which might cause people to use it ill-advisedly in inappropriate situations, but that's not the fault of PEP 470, and I don't want to see PEP 470 filled with warnings about how *other* uses of multi-index support might be inappropriate, because that will distract from the core message that is "we can fix the external hosting issue without needing clients to add a new mechanism". Paul From donald at stufft.io Wed Oct 8 13:07:15 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 8 Oct 2014 07:07:15 -0400 Subject: [Distutils] PEP470 installation security problems In-Reply-To: References: <20141003182800.GH7954@merlinux.eu> <20141007100925.GM7954@merlinux.eu> <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008100659.GQ7954@merlinux.eu> <20141008103300.GR7954@merlinux.eu> Message-ID: <49AB4EBA-037F-41DC-B409-0ACC739C9BD9@stufft.io> > On Oct 8, 2014, at 7:03 AM, Paul Moore wrote: > > On 8 October 2014 11:33, holger krekel wrote: >>> The use of --extra-index-url in >>> PEP 470 is to show how someone would add one of the extra repositories for a >>> project that is indexed on PyPI, which is again roughly as safe as installing >>> from PyPI at all. >> >> Then we are reading the sections i cite above very differently -- IMO >> you and the PEP generally push for multi-index ops without explaining >> the risks. >> >> Maybe someone else can chime in. > > Chiming in because you asked for other opinions, although I've not yet > read to the end of the thread... > > I read this section, and indeed the whole of the PEP, as basically saying: > > 1. We have a problem because PEP 438 didn't turn out so well in practice. > 2. We have an existing mechanism (multi-index support). > 3. The existing mechanism can be used as follows to better solve the > problem PEP 438 tried to solve. > > I don't see any "encouragement" to use multi-index support, other than > in the specific case PEP 438 was aimed at. Obviously PEP 470 raises > the profile of multi-index support, which might cause people to use it > ill-advisedly in inappropriate situations, but that's not the fault of > PEP 470, and I don't want to see PEP 470 filled with warnings about > how *other* uses of multi-index support might be inappropriate, > because that will distract from the core message that is "we can fix > the external hosting issue without needing clients to add a new > mechanism". > > Paul This is more or less exactly what I intend (and what I think it does) the PEP to say. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From ncoghlan at gmail.com Wed Oct 8 13:22:49 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 8 Oct 2014 21:22:49 +1000 Subject: [Distutils] PEP470 installation security problems In-Reply-To: <20141008105736.GS7954@merlinux.eu> References: <20141003182800.GH7954@merlinux.eu> <20141007100925.GM7954@merlinux.eu> <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> Message-ID: On 8 October 2014 20:57, holger krekel wrote: > On Wed, Oct 08, 2014 at 20:27 +1000, Nick Coghlan wrote: > Well, for installing NAME from pypi you need to trust that the people > who registered and maintain NAME are not doing something bad (and the > machine is not compromised but in that case all bets are off obviously). > And i can make a choice to trust "django", "flask, "warehouse" and other > pypi names. I am exposing myself to whatever the maintainers published > but it's my choice. This is a very different thing compared to: > > pip install --extra-index http://private.repo mypackage > > I may think i am trusting just "mypackage" from my private repo. > But in fact i am betting on nobody uploading "mypackage" to the pypi wiki. > I don't think this is very obvious to many -- it certainly wasn't > at EuroPython2014. So your concern is specifically with the fact that some users are not currently aware that "--extra-index" adds an *extra* index (which can then supply *any* package, as can the default index), and not a *replacement* index, and that they need to use --index-url in order to completely override the default index? Would you be more comfortable if the existing admonition in PEP 470 to use a private devpi instance with whitelisting in situations with a low security risk tolerance was accompanied by a concrete example that noted the appropriate option to use for private index URLs?: pip install --index-url private-repo.example.com mypackage Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From holger at merlinux.eu Wed Oct 8 13:40:49 2014 From: holger at merlinux.eu (holger krekel) Date: Wed, 8 Oct 2014 11:40:49 +0000 Subject: [Distutils] PEP470 installation security problems In-Reply-To: References: <20141007100925.GM7954@merlinux.eu> <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> Message-ID: <20141008114049.GU7954@merlinux.eu> On Wed, Oct 08, 2014 at 21:22 +1000, Nick Coghlan wrote: > On 8 October 2014 20:57, holger krekel wrote: > > On Wed, Oct 08, 2014 at 20:27 +1000, Nick Coghlan wrote: > > Well, for installing NAME from pypi you need to trust that the people > > who registered and maintain NAME are not doing something bad (and the > > machine is not compromised but in that case all bets are off obviously). > > And i can make a choice to trust "django", "flask, "warehouse" and other > > pypi names. I am exposing myself to whatever the maintainers published > > but it's my choice. This is a very different thing compared to: > > > > pip install --extra-index http://private.repo mypackage > > > > I may think i am trusting just "mypackage" from my private repo. > > But in fact i am betting on nobody uploading "mypackage" to the pypi wiki. > > I don't think this is very obvious to many -- it certainly wasn't > > at EuroPython2014. > > So your concern is specifically with the fact that some users are not > currently aware that "--extra-index" adds an *extra* index (which can > then supply *any* package, as can the default index), and not a > *replacement* index, and that they need to use --index-url in order to > completely override the default index? No, i am not concerned about the extra index supplying whatever packages. After all, the users specifies the option and should trust that index. I am concerned about the fact that public PyPI links are merged in even for my private packages residing on the extra index. > Would you be more comfortable if the existing admonition in PEP 470 to > use a private devpi instance with whitelisting in situations with a > low security risk tolerance was accompanied by a concrete example that > noted the appropriate option to use for private index URLs?: > > pip install --index-url private-repo.example.com mypackage I rather think the whole rationale "Why additional repositories?" section of the PEP needs a re-work and specifically not recommend --extra-index-url. Contrary to what Donald and Paul claim i don't see it discussing just the particular issue of using extra indexes for publically registered packages: http://legacy.python.org/dev/peps/pep-0470/#why-additional-repositories best, holger From p.f.moore at gmail.com Wed Oct 8 14:05:08 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 8 Oct 2014 13:05:08 +0100 Subject: [Distutils] PEP470 installation security problems In-Reply-To: <20141008114049.GU7954@merlinux.eu> References: <20141007100925.GM7954@merlinux.eu> <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> <20141008114049.GU7954@merlinux.eu> Message-ID: On 8 October 2014 12:40, holger krekel wrote: > I am concerned about the fact that public PyPI links are merged in even > for my private packages residing on the extra index. Bluntly, that's irrelevant. That's how pip works. Maybe it's not the best way, maybe a feature request for pip would be worth pursuing, maybe you could even argue that it's a security issue with pip. But it's not relevant to this PEP, which simply says that "for this *specific" problem, multi-index support is a viable solution". Asking for a change in behaviour from pip in this specific case is not what the PEP is about. Actually, pip's behaviour in general is not subject to the PEP process (as Donald pointed out, trying to make it be is what got PEP 438 in trouble). Paul From holger at merlinux.eu Wed Oct 8 14:17:38 2014 From: holger at merlinux.eu (holger krekel) Date: Wed, 8 Oct 2014 12:17:38 +0000 Subject: [Distutils] PEP470 installation security problems In-Reply-To: References: <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> <20141008114049.GU7954@merlinux.eu> Message-ID: <20141008121738.GV7954@merlinux.eu> On Wed, Oct 08, 2014 at 13:05 +0100, Paul Moore wrote: > On 8 October 2014 12:40, holger krekel wrote: > > I am concerned about the fact that public PyPI links are merged in even > > for my private packages residing on the extra index. > > Bluntly, that's irrelevant. I disagree. The PEP uses merging of public and private links in the main rationale section which comes before discussing migration strategies. It's used as motivation aka "look how easy it is to use additional/multi indexes" and not as a particular migration strategy that shouldn't be used otherwise. > That's how pip works. Maybe it's not the best way, maybe a feature > request for pip would be worth pursuing, maybe you could even argue > that it's a security issue with pip. But it's not relevant to this > PEP, which simply says that "for this *specific" problem, multi-index > support is a viable solution". Asking for a change in behaviour from > pip in this specific case is not what the PEP is about. Actually, > pip's behaviour in general is not subject to the PEP process (as > Donald pointed out, trying to make it be is what got PEP 438 in > trouble). Well, for one i think "--extra-index-url" is indeed broken UI exposing people to compromise without any warning. Also, i am worried on principle grounds if pip maintainers are putting themselves outside PEP reach, yet pip is distributed along with Python. best, holger From ncoghlan at gmail.com Wed Oct 8 14:18:35 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 8 Oct 2014 22:18:35 +1000 Subject: [Distutils] PEP470 installation security problems In-Reply-To: <20141008114049.GU7954@merlinux.eu> References: <20141007100925.GM7954@merlinux.eu> <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> <20141008114049.GU7954@merlinux.eu> Message-ID: On 8 October 2014 21:40, holger krekel wrote: > > No, i am not concerned about the extra index supplying whatever packages. > After all, the users specifies the option and should trust that index. > > I am concerned about the fact that public PyPI links are merged in even > for my private packages residing on the extra index. That's what a default repository *does*. It's always on, unless you explicitly turn it off. Hence the name *extra index*. The index URL option is the one to use if you want to *replace* the index. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Wed Oct 8 14:22:18 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 8 Oct 2014 08:22:18 -0400 Subject: [Distutils] PEP470 installation security problems In-Reply-To: <20141008121738.GV7954@merlinux.eu> References: <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> <20141008114049.GU7954@merlinux.eu> <20141008121738.GV7954@merlinux.eu> Message-ID: > On Oct 8, 2014, at 8:17 AM, holger krekel wrote: > > Also, i am worried on principle grounds if pip maintainers are putting > themselves outside PEP reach, yet pip is distributed along with Python. We?re not ?putting ourselves outside of PEP reach?. We are an external project and we are not bound by the PEP process. Devpi, py.test, Django, requests, etc are also not bound by the PEP process. I was worried this might be used to try and force pip to adhere to PEPs which is why PEP 453 explicitly mentions this fact. http://legacy.python.org/dev/peps/pep-0453/#policies-governance ?The maintainers of the bootstrapped software and the CPython core team will work together in order to address the needs of both. The bootstrapped software will still remain external to CPython and this PEP does not include CPython subsuming the development responsibilities or design decisions of the bootstrapped software. This PEP aims to decrease the burden on end users wanting to use third-party packages and the decisions inside it are pragmatic ones that represent the trust that the Python community has already placed in the Python Packaging Authority as the authors and maintainers of pip, setuptools, PyPI, virtualenv and other related projects.? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From ncoghlan at gmail.com Wed Oct 8 14:24:31 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 8 Oct 2014 22:24:31 +1000 Subject: [Distutils] PEP470 installation security problems In-Reply-To: <20141008121738.GV7954@merlinux.eu> References: <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> <20141008114049.GU7954@merlinux.eu> <20141008121738.GV7954@merlinux.eu> Message-ID: On 8 October 2014 22:17, holger krekel wrote: > On Wed, Oct 08, 2014 at 13:05 +0100, Paul Moore wrote: >> On 8 October 2014 12:40, holger krekel wrote: >> > I am concerned about the fact that public PyPI links are merged in even >> > for my private packages residing on the extra index. >> >> Bluntly, that's irrelevant. > > I disagree. The PEP uses merging of public and private links in > the main rationale section which comes before discussing migration > strategies. It's used as motivation aka "look how easy it is > to use additional/multi indexes" and not as a particular migration > strategy that shouldn't be used otherwise. OK, I think I understand your concern now - the PEP includes an example of a practice that you don't like and would prefer to see strongly discouraged. We can just delete all references to private indexes from the PEP, as they were merely included as an illustration of one of the reasons the multi-index/alternative-index support already exists. If you find the example distracting from the actual point of the PEP, then the example isn't serving its purpose, and we're better off without it. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed Oct 8 14:30:54 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 8 Oct 2014 22:30:54 +1000 Subject: [Distutils] PEP470 installation security problems In-Reply-To: References: <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> <20141008114049.GU7954@merlinux.eu> <20141008121738.GV7954@merlinux.eu> Message-ID: On 8 October 2014 22:22, Donald Stufft wrote: > >> On Oct 8, 2014, at 8:17 AM, holger krekel wrote: >> >> Also, i am worried on principle grounds if pip maintainers are putting >> themselves outside PEP reach, yet pip is distributed along with Python. > > We?re not ?putting ourselves outside of PEP reach?. We are an external > project and we are not bound by the PEP process. Devpi, py.test, Django, > requests, etc are also not bound by the PEP process. Note also that even for CPython itself, it is *up to us as core developers* to decide when something needs to be escalated through the PEP process. The vast majority of CPython changes are handled directly through the issue tracker, and there's still the occasional change that doesn't even make it that far (e.g. if we notice a problem while working on something else, we have the option of just committing the fix directly). PEPs are primarily for changes which have broad ecosystem implications where the additional overhead is justified. We don't write PEPs for every change to the CPython command line interface (e.g. there's no PEP for isolated mode), and the same kind of assessment of external impact applies to pip and the PyPA in general when decided whether a change can be handled within the scope of an individual project, or if it needs to be escalated for broader discussion. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Wed Oct 8 14:37:31 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 8 Oct 2014 08:37:31 -0400 Subject: [Distutils] PEP470 installation security problems In-Reply-To: References: <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> <20141008114049.GU7954@merlinux.eu> <20141008121738.GV7954@merlinux.eu> Message-ID: > On Oct 8, 2014, at 8:24 AM, Nick Coghlan wrote: > > On 8 October 2014 22:17, holger krekel wrote: >> On Wed, Oct 08, 2014 at 13:05 +0100, Paul Moore wrote: >>> On 8 October 2014 12:40, holger krekel wrote: >>>> I am concerned about the fact that public PyPI links are merged in even >>>> for my private packages residing on the extra index. >>> >>> Bluntly, that's irrelevant. >> >> I disagree. The PEP uses merging of public and private links in >> the main rationale section which comes before discussing migration >> strategies. It's used as motivation aka "look how easy it is >> to use additional/multi indexes" and not as a particular migration >> strategy that shouldn't be used otherwise. > > OK, I think I understand your concern now - the PEP includes an > example of a practice that you don't like and would prefer to see > strongly discouraged. Does it? The only examples in the PEP are showing: A) How can I, as an author of a project who wishes to use this new mechanism do so for my project. B) How can I, as a user of a project who is using this new mechanism tell pip to add this additional *public* repository so that I can install it since they don?t host on PyPI. > > We can just delete all references to private indexes from the PEP, as > they were merely included as an illustration of one of the reasons the > multi-index/alternative-index support already exists. If you find the > example distracting from the actual point of the PEP, then the example > isn't serving its purpose, and we're better off without it. > There is really only one mention in the entire PEP that I can remember or find in a quick re-skim. That is in: "Additionally, the multiple repository approach is a concept that is useful outside of the narrow scope of allowing projects which wish to be included on the index portion of PyPI but do not wish to utilize the repository portion of PyPI. This includes places where a company may wish to host a repository that contains their internal packages or where a project may wish to have multiple "channels" of releases, such as alpha, beta, release candidate, and final release.? Which is just saying ?hey this concept of pip works with repositories, not PyPI, PyPI just happens to be the default repository? not only already exists, but is useful in more situations and one of those situations is internal company repositories. I can remove the half a dozen total words that constitute the only reference in the PEP to a private anything, but I?m still confused how this somehow correlates to the PEP is advocating everyone switch to using ?extra-index-url for their private repositories when in reality the PEP is giving and example of what someone would need to do, as pip currently stands, to utilize a project that uses this feature. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From holger at merlinux.eu Wed Oct 8 14:43:55 2014 From: holger at merlinux.eu (holger krekel) Date: Wed, 8 Oct 2014 12:43:55 +0000 Subject: [Distutils] PEP470 installation security problems In-Reply-To: References: <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> <20141008114049.GU7954@merlinux.eu> Message-ID: <20141008124355.GX7954@merlinux.eu> On Wed, Oct 08, 2014 at 22:18 +1000, Nick Coghlan wrote: > On 8 October 2014 21:40, holger krekel wrote: > > > > No, i am not concerned about the extra index supplying whatever packages. > > After all, the users specifies the option and should trust that index. > > > > I am concerned about the fact that public PyPI links are merged in even > > for my private packages residing on the extra index. > > That's what a default repository *does*. It's always on, unless you > explicitly turn it off. Hence the name *extra index*. The index URL > option is the one to use if you want to *replace* the index. Nick, i don't know why you are saying this. Do you think i don't know this? My point is that PyPI makes for a very different default repository than the Debian or Redhat one. Or do you disagree there? holger From donald at stufft.io Wed Oct 8 14:47:05 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 8 Oct 2014 08:47:05 -0400 Subject: [Distutils] PEP470 installation security problems In-Reply-To: <20141008124355.GX7954@merlinux.eu> References: <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> <20141008114049.GU7954@merlinux.eu> <20141008124355.GX7954@merlinux.eu> Message-ID: <98E30675-9A38-41A1-A23D-8501B88BC8A3@stufft.io> > On Oct 8, 2014, at 8:43 AM, holger krekel wrote: > > On Wed, Oct 08, 2014 at 22:18 +1000, Nick Coghlan wrote: >> On 8 October 2014 21:40, holger krekel wrote: >>> >>> No, i am not concerned about the extra index supplying whatever packages. >>> After all, the users specifies the option and should trust that index. >>> >>> I am concerned about the fact that public PyPI links are merged in even >>> for my private packages residing on the extra index. >> >> That's what a default repository *does*. It's always on, unless you >> explicitly turn it off. Hence the name *extra index*. The index URL >> option is the one to use if you want to *replace* the index. > > Nick, i don't know why you are saying this. Do you think i don't know this? > > My point is that PyPI makes for a very different default repository than the > Debian or Redhat one. Or do you disagree there? If you understand that, then your statements in here don?t make any sense to me. What is it you?re trying to achieve exactly? Do you think the PEP should be rejected? Do you think it needs amended? You?re saying things that I can?t reconcile how they relate to the PEP (and I?m apparently not the only one) nor can I convert them into actionable feedback. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From mal at egenix.com Wed Oct 8 14:55:48 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 08 Oct 2014 14:55:48 +0200 Subject: [Distutils] PEP470 installation security problems In-Reply-To: References: <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> <20141008114049.GU7954@merlinux.eu> <20141008121738.GV7954@merlinux.eu> Message-ID: <54353454.206@egenix.com> On 08.10.2014 14:30, Nick Coghlan wrote: > On 8 October 2014 22:22, Donald Stufft wrote: >> >>> On Oct 8, 2014, at 8:17 AM, holger krekel wrote: >>> >>> Also, i am worried on principle grounds if pip maintainers are putting >>> themselves outside PEP reach, yet pip is distributed along with Python. >> >> We?re not ?putting ourselves outside of PEP reach?. We are an external >> project and we are not bound by the PEP process. Devpi, py.test, Django, >> requests, etc are also not bound by the PEP process. > > Note also that even for CPython itself, it is *up to us as core > developers* to decide when something needs to be escalated through the > PEP process. The vast majority of CPython changes are handled directly > through the issue tracker, and there's still the occasional change > that doesn't even make it that far (e.g. if we notice a problem while > working on something else, we have the option of just committing the > fix directly). > > PEPs are primarily for changes which have broad ecosystem implications > where the additional overhead is justified. We don't write PEPs for > every change to the CPython command line interface (e.g. there's no > PEP for isolated mode), and the same kind of assessment of external > impact applies to pip and the PyPA in general when decided whether a > change can be handled within the scope of an individual project, or if > it needs to be escalated for broader discussion. I don't follow Donald's reasoning and I'm not sure I understand whether your comments are meant as clarification of pip being subject to the PEP process or support for Donald's reasoning :-) Changes to pip and PyPI *do* have a global effect on the Python ecosystem and thus need to be covered by the PEP process. If pip decides to go with a strategy that ignores this, I think we have a problem. The core developers put trust into pip when allowing it to (effectively) get distributed with Python and making it the default Python packaging manager. Please use that trust with the appropriate care and respect. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From holger at merlinux.eu Wed Oct 8 14:59:36 2014 From: holger at merlinux.eu (holger krekel) Date: Wed, 8 Oct 2014 12:59:36 +0000 Subject: [Distutils] PEP470 installation security problems In-Reply-To: <98E30675-9A38-41A1-A23D-8501B88BC8A3@stufft.io> References: <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> <20141008114049.GU7954@merlinux.eu> <20141008124355.GX7954@merlinux.eu> <98E30675-9A38-41A1-A23D-8501B88BC8A3@stufft.io> Message-ID: <20141008125936.GY7954@merlinux.eu> On Wed, Oct 08, 2014 at 08:47 -0400, Donald Stufft wrote: > > On Oct 8, 2014, at 8:43 AM, holger krekel wrote: > > > > On Wed, Oct 08, 2014 at 22:18 +1000, Nick Coghlan wrote: > >> On 8 October 2014 21:40, holger krekel wrote: > >>> > >>> No, i am not concerned about the extra index supplying whatever packages. > >>> After all, the users specifies the option and should trust that index. > >>> > >>> I am concerned about the fact that public PyPI links are merged in even > >>> for my private packages residing on the extra index. > >> > >> That's what a default repository *does*. It's always on, unless you > >> explicitly turn it off. Hence the name *extra index*. The index URL > >> option is the one to use if you want to *replace* the index. > > > > Nick, i don't know why you are saying this. Do you think i don't know this? > > > > My point is that PyPI makes for a very different default repository than the > > Debian or Redhat one. Or do you disagree there? > > If you understand that, then your statements in here don?t make any sense to me. > > What is it you?re trying to achieve exactly? Do you think the PEP should be > rejected? Do you think it needs amended? You?re saying things that I can?t reconcile > how they relate to the PEP (and I?m apparently not the only one) nor can I convert > them into actionable feedback. Sorry that it's so unclear to you, Nick and Paul. I tried my best. And i tried to make suggestions what to change, what to avoid, what kind of options pip would need to become safer etc.. That was all meant as useful feedback to get a better PEP and end result. But if you and Nick as authors refuse my suggestions (mainly: backward compat, more careful reasoning about multi-index ops) then i am currently clearly -1 on the PEP because i think it does more harm than good. And i'll let it all rest at that for a bit because i don't want to spend more time on it right now. best, holger From donald at stufft.io Wed Oct 8 15:05:20 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 8 Oct 2014 09:05:20 -0400 Subject: [Distutils] PEP470 installation security problems In-Reply-To: <54353454.206@egenix.com> References: <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> <20141008114049.GU7954@merlinux.eu> <20141008121738.GV7954@merlinux.eu> <54353454.206@egenix.com> Message-ID: <67501766-E761-4E11-859F-93233B8AF9B8@stufft.io> > On Oct 8, 2014, at 8:55 AM, M.-A. Lemburg wrote: > > On 08.10.2014 14:30, Nick Coghlan wrote: >> On 8 October 2014 22:22, Donald Stufft wrote: >>> >>>> On Oct 8, 2014, at 8:17 AM, holger krekel wrote: >>>> >>>> Also, i am worried on principle grounds if pip maintainers are putting >>>> themselves outside PEP reach, yet pip is distributed along with Python. >>> >>> We?re not ?putting ourselves outside of PEP reach?. We are an external >>> project and we are not bound by the PEP process. Devpi, py.test, Django, >>> requests, etc are also not bound by the PEP process. >> >> Note also that even for CPython itself, it is *up to us as core >> developers* to decide when something needs to be escalated through the >> PEP process. The vast majority of CPython changes are handled directly >> through the issue tracker, and there's still the occasional change >> that doesn't even make it that far (e.g. if we notice a problem while >> working on something else, we have the option of just committing the >> fix directly). >> >> PEPs are primarily for changes which have broad ecosystem implications >> where the additional overhead is justified. We don't write PEPs for >> every change to the CPython command line interface (e.g. there's no >> PEP for isolated mode), and the same kind of assessment of external >> impact applies to pip and the PyPA in general when decided whether a >> change can be handled within the scope of an individual project, or if >> it needs to be escalated for broader discussion. > > I don't follow Donald's reasoning and I'm not sure I understand > whether your comments are meant as clarification of pip being > subject to the PEP process or support for Donald's reasoning :-) > > Changes to pip and PyPI *do* have a global effect on the Python > ecosystem and thus need to be covered by the PEP process. > > If pip decides to go with a strategy that ignores this, I think we > have a problem. The core developers put trust into pip when allowing > it to (effectively) get distributed with Python and making it the > default Python packaging manager. Please use that trust with the > appropriate care and respect. I don?t think we?ve *ever* not used that trust with care and respect and we?ve been trusted by the Python community for far longer than PEP 453 has existed. We attempt to follow PEPs where we can and where they make good sense. Nobody on the pip team is saying we?re going to flat out ignore PEPs or whatever. We (or at least I am) are saying that dictating UX via PEP process has been shown to us *not* to work and that we are not obligated to implement or listen to a PEP. This was explicitly spelled out in PEP 453 that we remain an external project even with the fact we?re now bundled with Python. This does not mean we won?t generally try to use the PEP process where our changes have cross cutting concerns between different projects but it does mean that we implement or follow PEPs at our discretion. This isn?t up for debate, it was an explicit inclusion in PEP 453 and if there was a problem with pip maintaining it?s own project the time to bring that up was a year ago. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From donald at stufft.io Wed Oct 8 15:13:59 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 8 Oct 2014 09:13:59 -0400 Subject: [Distutils] PEP470 installation security problems In-Reply-To: <20141008125936.GY7954@merlinux.eu> References: <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> <20141008114049.GU7954@merlinux.eu> <20141008124355.GX7954@merlinux.eu> <98E30675-9A38-41A1-A23D-8501B88BC8A3@stufft.io> <20141008125936.GY7954@merlinux.eu> Message-ID: <737F43B0-7009-4DDE-BBAD-2BA4F48D224D@stufft.io> > On Oct 8, 2014, at 8:59 AM, holger krekel wrote: > > On Wed, Oct 08, 2014 at 08:47 -0400, Donald Stufft wrote: >>> On Oct 8, 2014, at 8:43 AM, holger krekel wrote: >>> >>> On Wed, Oct 08, 2014 at 22:18 +1000, Nick Coghlan wrote: >>>> On 8 October 2014 21:40, holger krekel wrote: >>>>> >>>>> No, i am not concerned about the extra index supplying whatever packages. >>>>> After all, the users specifies the option and should trust that index. >>>>> >>>>> I am concerned about the fact that public PyPI links are merged in even >>>>> for my private packages residing on the extra index. >>>> >>>> That's what a default repository *does*. It's always on, unless you >>>> explicitly turn it off. Hence the name *extra index*. The index URL >>>> option is the one to use if you want to *replace* the index. >>> >>> Nick, i don't know why you are saying this. Do you think i don't know this? >>> >>> My point is that PyPI makes for a very different default repository than the >>> Debian or Redhat one. Or do you disagree there? >> >> If you understand that, then your statements in here don?t make any sense to me. >> >> What is it you?re trying to achieve exactly? Do you think the PEP should be >> rejected? Do you think it needs amended? You?re saying things that I can?t reconcile >> how they relate to the PEP (and I?m apparently not the only one) nor can I convert >> them into actionable feedback. > > Sorry that it's so unclear to you, Nick and Paul. I tried my best. > And i tried to make suggestions what to change, what to avoid, what > kind of options pip would need to become safer etc.. That was all meant > as useful feedback to get a better PEP and end result. > > But if you and Nick as authors refuse my suggestions (mainly: > backward compat, more careful reasoning about multi-index ops) then i am > currently clearly -1 on the PEP because i think it does more harm than good. > > And i'll let it all rest at that for a bit because i don't want to > spend more time on it right now. I think I responded why I had considered and then rejected the backwards compatibility concern. We may just disagree on that point. I don?t understand what ?more careful reasoning about multi-index ops? means. Maybe if you suggest a rewording or point to a specific part of the PEP that you think should be removed/edited/added to? If you?d rather not do that above that?s fine! Just saying if you care to spend more time on it that maybe an explicit suggestion of what to change in the PEP would be easier to understand. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From p.f.moore at gmail.com Wed Oct 8 15:15:29 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 8 Oct 2014 14:15:29 +0100 Subject: [Distutils] PEP470 installation security problems In-Reply-To: <54353454.206@egenix.com> References: <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> <20141008114049.GU7954@merlinux.eu> <20141008121738.GV7954@merlinux.eu> <54353454.206@egenix.com> Message-ID: On 8 October 2014 13:55, M.-A. Lemburg wrote: > If pip decides to go with a strategy that ignores this, I think we > have a problem. The core developers put trust into pip when allowing > it to (effectively) get distributed with Python and making it the > default Python packaging manager. Please use that trust with the > appropriate care and respect. Just to clarify - the pip team (I hope I speak for all of us) fully understand the implications of being the de facto standard package manager. And we appreciate the trust placed in us by the fact that pip is distributed with Python. But at the same time, that trust was given on the basis that (presumably) we have a track record of doing things right, in an area that is notoriously full of heated discussions and conflicting opinions. So what we'd like to do is to continue handling things in the same way as always, working with the packaging community. In particular, that means that we did not align ourselves to the CPython development model (as it is designed for a very different community and set of problems). But we do want to adopt their good practices where possible and appropriate. One of those is the PEP process - but it's not entirely suitable (see the trail of PEPs from the distribute/packaging/distutils2 era, for why). So we're trying to get things right, and in the process we're learning - for example, the failure of PEP 438 taught us that specifying installer behaviour too closely in a PEP means we can't fix problems that are completely messing up our users. But we still believe in the PEP process (anyone who thinks otherwise hasn't noticed the amount of effort Donald, in particular, is putting into all the PEPs in progress). It doesn't mean that it can be treated as a way of forcing us not to do what we think is right for the pip user base, though. Paul. From p.f.moore at gmail.com Wed Oct 8 15:27:13 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 8 Oct 2014 14:27:13 +0100 Subject: [Distutils] PEP470 installation security problems In-Reply-To: <20141008125936.GY7954@merlinux.eu> References: <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> <20141008114049.GU7954@merlinux.eu> <20141008124355.GX7954@merlinux.eu> <98E30675-9A38-41A1-A23D-8501B88BC8A3@stufft.io> <20141008125936.GY7954@merlinux.eu> Message-ID: On 8 October 2014 13:59, holger krekel wrote: > But if you and Nick as authors refuse my suggestions (mainly: > backward compat, more careful reasoning about multi-index ops) then i am > currently clearly -1 on the PEP because i think it does more harm than good. Holger, there's been a lot said in this thread, and it's entirely possible I may have missed something crucial. But it seems to me that a lot of the debate has been about wording and rationale. Can I just cross-check with you, before you leave the discussion: 1. Ignoring all of the explanations and rationale, are you -1 on the technical changes being proposed? 2. Do you have an alternative proposal, or is your -1 in effect a vote to do nothing? Personally, I think that we have to do something about the pip user interface, as the current situation is harming our users. If PEP 470 isn't accepted, we'll need to look again at what we do in relation to PEP 438 support. Frankly, I'd rather not go there, as I think it's clear from the feedback we've received that full support is harmful to our users. Paul From mal at egenix.com Wed Oct 8 15:40:25 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 08 Oct 2014 15:40:25 +0200 Subject: [Distutils] PEP470 installation security problems In-Reply-To: <67501766-E761-4E11-859F-93233B8AF9B8@stufft.io> References: <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> <20141008114049.GU7954@merlinux.eu> <20141008121738.GV7954@merlinux.eu> <54353454.206@egenix.com> <67501766-E761-4E11-859F-93233B8AF9B8@stufft.io> Message-ID: <54353EC9.1040608@egenix.com> On 08.10.2014 15:05, Donald Stufft wrote: > >> On Oct 8, 2014, at 8:55 AM, M.-A. Lemburg wrote: >> >> On 08.10.2014 14:30, Nick Coghlan wrote: >>> On 8 October 2014 22:22, Donald Stufft wrote: >>>> >>>>> On Oct 8, 2014, at 8:17 AM, holger krekel wrote: >>>>> >>>>> Also, i am worried on principle grounds if pip maintainers are putting >>>>> themselves outside PEP reach, yet pip is distributed along with Python. >>>> >>>> We?re not ?putting ourselves outside of PEP reach?. We are an external >>>> project and we are not bound by the PEP process. Devpi, py.test, Django, >>>> requests, etc are also not bound by the PEP process. >>> >>> Note also that even for CPython itself, it is *up to us as core >>> developers* to decide when something needs to be escalated through the >>> PEP process. The vast majority of CPython changes are handled directly >>> through the issue tracker, and there's still the occasional change >>> that doesn't even make it that far (e.g. if we notice a problem while >>> working on something else, we have the option of just committing the >>> fix directly). >>> >>> PEPs are primarily for changes which have broad ecosystem implications >>> where the additional overhead is justified. We don't write PEPs for >>> every change to the CPython command line interface (e.g. there's no >>> PEP for isolated mode), and the same kind of assessment of external >>> impact applies to pip and the PyPA in general when decided whether a >>> change can be handled within the scope of an individual project, or if >>> it needs to be escalated for broader discussion. >> >> I don't follow Donald's reasoning and I'm not sure I understand >> whether your comments are meant as clarification of pip being >> subject to the PEP process or support for Donald's reasoning :-) >> >> Changes to pip and PyPI *do* have a global effect on the Python >> ecosystem and thus need to be covered by the PEP process. >> >> If pip decides to go with a strategy that ignores this, I think we >> have a problem. The core developers put trust into pip when allowing >> it to (effectively) get distributed with Python and making it the >> default Python packaging manager. Please use that trust with the >> appropriate care and respect. > > I don?t think we?ve *ever* not used that trust with care and respect and > we?ve been trusted by the Python community for far longer than PEP 453 > has existed. We attempt to follow PEPs where we can and where they make > good sense. Nobody on the pip team is saying we?re going to flat out > ignore PEPs or whatever. > > We (or at least I am) are saying that dictating UX via PEP process has > been shown to us *not* to work and that we are not obligated to implement > or listen to a PEP. This was explicitly spelled out in PEP 453 that we > remain an external project even with the fact we?re now bundled with > Python. This does not mean we won?t generally try to use the PEP process > where our changes have cross cutting concerns between different projects > but it does mean that we implement or follow PEPs at our discretion. This > isn?t up for debate, it was an explicit inclusion in PEP 453 and if there > was a problem with pip maintaining it?s own project the time to bring that > up was a year ago. The intention of PEP 435 was to enable pip to evolve independent of the Python release process, which is a good thing. However, your comment that "We are an external project and we are not bound by the PEP process." doesn't really pan out in the light of the PEP's requirement that "The maintainers of the bootstrapped software and the CPython core team will work together in order to address the needs of both." If pip maintainers don't feel they are bound by PEPs, you could argue that you are also not bound by PEP 435, which would result in a rather pointless cooperation setup :-) Note that I'm not trying to say that you are actually not respecting the PEP process. I'm just concerned about comments like the above causing unnecessary heat in discussions. I'd also like to request that you take Holger's concerns more seriously, perhaps add him as PEP author and let him participate in clarifying it (if he still feels like investing time in this). PEPs are never perfect and there's always room for improvement. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Wed Oct 8 15:59:55 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 8 Oct 2014 23:59:55 +1000 Subject: [Distutils] PEP470 installation security problems In-Reply-To: <54353EC9.1040608@egenix.com> References: <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> <20141008114049.GU7954@merlinux.eu> <20141008121738.GV7954@merlinux.eu> <54353454.206@egenix.com> <67501766-E761-4E11-859F-93233B8AF9B8@stufft.io> <54353EC9.1040608@egenix.com> Message-ID: On 8 Oct 2014 23:40, "M.-A. Lemburg" wrote: > > The intention of PEP 435 was to enable pip to evolve independent > of the Python release process, which is a good thing. > > However, your comment that "We are an external project and we are not > bound by the PEP process." doesn't really pan out in the light of the PEP's > requirement that "The maintainers of the bootstrapped software and the > CPython core team will work together in order to address the needs of both." > > If pip maintainers don't feel they are bound by PEPs, you could argue > that you are also not bound by PEP 435, which would result in a > rather pointless cooperation setup :-) > > Note that I'm not trying to say that you are actually not respecting the > PEP process. I'm just concerned about comments like the above causing > unnecessary heat in discussions. pip's UX decisions aren't likely to ever be put through the PEP process again - the PEP 426 (and now PEP 470) model of providing functional requirements and recommendations in the form of MUST and SHOULD statements is a cleaner process, since they provide guidance for all clients, not just pip, and leave the *details* of the UX to the normal pip development cycle (so if user feedback indicates a problem with the specifics of the initial approach, they can address that while remaining compliant with the specification). Decoupling functional specifications from UX details of individual tools is a good idea in general, this is just applying that model to pip and the PEP process in particular. PyPI needs to be covered in more detail, however, as these PEPs also serve as the *interface* specification for both clients and servers, and those need concrete API definitions to work with. PEP 438 was the main case so far where the PEP included specific UX design details for pip, and that's the aspect that *won't* be repeated. Regards, Nick. > > I'd also like to request that you take Holger's concerns more > seriously, perhaps add him as PEP author and let him participate > in clarifying it (if he still feels like investing time in this). > > PEPs are never perfect and there's always room for improvement. > > Thanks, > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source > >>> Python/Zope Consulting and Support ... http://www.egenix.com/ > >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: > > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Oct 8 16:04:59 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 8 Oct 2014 10:04:59 -0400 Subject: [Distutils] PEP470 installation security problems In-Reply-To: <54353EC9.1040608@egenix.com> References: <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> <20141008114049.GU7954@merlinux.eu> <20141008121738.GV7954@merlinux.eu> <54353454.206@egenix.com> <67501766-E761-4E11-859F-93233B8AF9B8@stufft.io> <54353EC9.1040608@egenix.com> Message-ID: <21FCC295-4B82-47E6-9997-83478E695D24@stufft.io> > On Oct 8, 2014, at 9:40 AM, M.-A. Lemburg wrote: > > On 08.10.2014 15:05, Donald Stufft wrote: >> >>> On Oct 8, 2014, at 8:55 AM, M.-A. Lemburg wrote: >>> >>> On 08.10.2014 14:30, Nick Coghlan wrote: >>>> On 8 October 2014 22:22, Donald Stufft wrote: >>>>> >>>>>> On Oct 8, 2014, at 8:17 AM, holger krekel wrote: >>>>>> >>>>>> Also, i am worried on principle grounds if pip maintainers are putting >>>>>> themselves outside PEP reach, yet pip is distributed along with Python. >>>>> >>>>> We?re not ?putting ourselves outside of PEP reach?. We are an external >>>>> project and we are not bound by the PEP process. Devpi, py.test, Django, >>>>> requests, etc are also not bound by the PEP process. >>>> >>>> Note also that even for CPython itself, it is *up to us as core >>>> developers* to decide when something needs to be escalated through the >>>> PEP process. The vast majority of CPython changes are handled directly >>>> through the issue tracker, and there's still the occasional change >>>> that doesn't even make it that far (e.g. if we notice a problem while >>>> working on something else, we have the option of just committing the >>>> fix directly). >>>> >>>> PEPs are primarily for changes which have broad ecosystem implications >>>> where the additional overhead is justified. We don't write PEPs for >>>> every change to the CPython command line interface (e.g. there's no >>>> PEP for isolated mode), and the same kind of assessment of external >>>> impact applies to pip and the PyPA in general when decided whether a >>>> change can be handled within the scope of an individual project, or if >>>> it needs to be escalated for broader discussion. >>> >>> I don't follow Donald's reasoning and I'm not sure I understand >>> whether your comments are meant as clarification of pip being >>> subject to the PEP process or support for Donald's reasoning :-) >>> >>> Changes to pip and PyPI *do* have a global effect on the Python >>> ecosystem and thus need to be covered by the PEP process. >>> >>> If pip decides to go with a strategy that ignores this, I think we >>> have a problem. The core developers put trust into pip when allowing >>> it to (effectively) get distributed with Python and making it the >>> default Python packaging manager. Please use that trust with the >>> appropriate care and respect. >> >> I don?t think we?ve *ever* not used that trust with care and respect and >> we?ve been trusted by the Python community for far longer than PEP 453 >> has existed. We attempt to follow PEPs where we can and where they make >> good sense. Nobody on the pip team is saying we?re going to flat out >> ignore PEPs or whatever. >> >> We (or at least I am) are saying that dictating UX via PEP process has >> been shown to us *not* to work and that we are not obligated to implement >> or listen to a PEP. This was explicitly spelled out in PEP 453 that we >> remain an external project even with the fact we?re now bundled with >> Python. This does not mean we won?t generally try to use the PEP process >> where our changes have cross cutting concerns between different projects >> but it does mean that we implement or follow PEPs at our discretion. This >> isn?t up for debate, it was an explicit inclusion in PEP 453 and if there >> was a problem with pip maintaining it?s own project the time to bring that >> up was a year ago. > > The intention of PEP 435 was to enable pip to evolve independent > of the Python release process, which is a good thing. > > However, your comment that "We are an external project and we are not > bound by the PEP process." doesn't really pan out in the light of the PEP's > requirement that "The maintainers of the bootstrapped software and the > CPython core team will work together in order to address the needs of both." > > If pip maintainers don't feel they are bound by PEPs, you could argue > that you are also not bound by PEP 435, which would result in a > rather pointless cooperation setup :-) > > Note that I'm not trying to say that you are actually not respecting the > PEP process. I'm just concerned about comments like the above causing > unnecessary heat in discussions. I feel like this whole ?Is pip subject to PEPs? thing went way off the rails somewhere. Originally it was just ?A PEP can?t mandate to an installer? which is true, pip is the only installer bundled with Python and I try to write my PEPs to be installer agnostic. I think it also got bound up in the fact that I/we feel pretty strongly that dictating a UX in a PEP to pip doesn?t work (discovered through experience with PEP 438) and thus we?re unlikely to listen to a PEP that dictates a UX that we feel is bad again because it?s turned out to be bad idea for us (although it?s more likely that such a PEP wouldn?t get accepted to begin with I think). Somewhere that morphed to pip is not subject to PEPs, I think with the suggestion that the changes Holger is asking to be made are specific to the pip functionality and not to PEP 470 so he should raise those issues on the pip tracker as they aren?t part of PEP 470 and that pip doesn?t generally use the PEP process for our behavior). From there is snowballed into this argument which I think is likely to just be circular and largely pointless as I don?t think a situation where pip flat out refuses to follow a reasonable PEP (or even a request from python-dev) is likely to come up so the difference is academic. > > I'd also like to request that you take Holger's concerns more > seriously, perhaps add him as PEP author and let him participate > in clarifying it (if he still feels like investing time in this). I take all concerns and feedback seriously else I wouldn?t spend the many hours I?ve spent just this morning responding to them. I don?t grok what Holger?s actual concern is so it?s hard to turn those concerns into anything actionable I can actually do on the PEP. I?ve asked for him (if he desires!) to give an actual example of something he?d change in the PEP to see if maybe that would make it clearer to me what he?s actually concerned about in relation to the PEP. I?m going to remove the one half a sentence that mentions a private repository in any capacity but I have a hard time believing that using it as one example in a list is what Holger?s concern is so I feel like that probably won?t fully address it. > > PEPs are never perfect and there's always room for improvement. > > Thanks, > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source >>>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: > > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > http://www.egenix.com/company/contact/ --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From chris.jerdonek at gmail.com Wed Oct 8 17:08:15 2014 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Wed, 8 Oct 2014 08:08:15 -0700 Subject: [Distutils] PEP470 installation security problems In-Reply-To: <737F43B0-7009-4DDE-BBAD-2BA4F48D224D@stufft.io> References: <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> <20141008114049.GU7954@merlinux.eu> <20141008124355.GX7954@merlinux.eu> <98E30675-9A38-41A1-A23D-8501B88BC8A3@stufft.io> <20141008125936.GY7954@merlinux.eu> <737F43B0-7009-4DDE-BBAD-2BA4F48D224D@stufft.io> Message-ID: I have a suggestion. Holger obviously feels he has something very important to say, and a lot of e-mails have already been sent back and forth. Is there some way that Donald, Nick, and Holger could perhaps have a conference call or hangout of some sort just for the purpose of understanding and/or confirming exactly what his concern is (and, if possible, coming to agreement on a resolution)? And then the result of that conversation can be summarized for the list? I think that might be more constructive at this point and courteous to Holger. I know that for me, sometimes "a quick phone call" can do wonders. --Chris On Wed, Oct 8, 2014 at 6:13 AM, Donald Stufft wrote: > > > On Oct 8, 2014, at 8:59 AM, holger krekel wrote: > > > > On Wed, Oct 08, 2014 at 08:47 -0400, Donald Stufft wrote: > >>> On Oct 8, 2014, at 8:43 AM, holger krekel wrote: > >>> > >>> On Wed, Oct 08, 2014 at 22:18 +1000, Nick Coghlan wrote: > >>>> On 8 October 2014 21:40, holger krekel wrote: > >>>>> > >>>>> No, i am not concerned about the extra index supplying whatever > packages. > >>>>> After all, the users specifies the option and should trust that > index. > >>>>> > >>>>> I am concerned about the fact that public PyPI links are merged in > even > >>>>> for my private packages residing on the extra index. > >>>> > >>>> That's what a default repository *does*. It's always on, unless you > >>>> explicitly turn it off. Hence the name *extra index*. The index URL > >>>> option is the one to use if you want to *replace* the index. > >>> > >>> Nick, i don't know why you are saying this. Do you think i don't know > this? > >>> > >>> My point is that PyPI makes for a very different default repository > than the > >>> Debian or Redhat one. Or do you disagree there? > >> > >> If you understand that, then your statements in here don?t make any > sense to me. > >> > >> What is it you?re trying to achieve exactly? Do you think the PEP > should be > >> rejected? Do you think it needs amended? You?re saying things that I > can?t reconcile > >> how they relate to the PEP (and I?m apparently not the only one) nor > can I convert > >> them into actionable feedback. > > > > Sorry that it's so unclear to you, Nick and Paul. I tried my best. > > And i tried to make suggestions what to change, what to avoid, what > > kind of options pip would need to become safer etc.. That was all meant > > as useful feedback to get a better PEP and end result. > > > > But if you and Nick as authors refuse my suggestions (mainly: > > backward compat, more careful reasoning about multi-index ops) then i am > > currently clearly -1 on the PEP because i think it does more harm than > good. > > > > And i'll let it all rest at that for a bit because i don't want to > > spend more time on it right now. > > I think I responded why I had considered and then rejected the backwards > compatibility concern. We may just disagree on that point. > > I don?t understand what ?more careful reasoning about multi-index ops? > means. > Maybe if you suggest a rewording or point to a specific part of the PEP > that > you think should be removed/edited/added to? > > If you?d rather not do that above that?s fine! Just saying if you care to > spend > more time on it that maybe an explicit suggestion of what to change in the > PEP > would be easier to understand. > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Wed Oct 8 19:54:22 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 08 Oct 2014 19:54:22 +0200 Subject: [Distutils] PEP470 installation security problems In-Reply-To: References: <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> <20141008114049.GU7954@merlinux.eu> <20141008121738.GV7954@merlinux.eu> <54353454.206@egenix.com> <67501766-E761-4E11-859F-93233B8AF9B8@stufft.io> <54353EC9.1040608@egenix.com> Message-ID: <54357A4E.5080108@egenix.com> On 08.10.2014 15:59, Nick Coghlan wrote: > On 8 Oct 2014 23:40, "M.-A. Lemburg" wrote: >> The intention of PEP 435 was to enable pip to evolve independent >> of the Python release process, which is a good thing. >> >> However, your comment that "We are an external project and we are not >> bound by the PEP process." doesn't really pan out in the light of the PEP's >> requirement that "The maintainers of the bootstrapped software and the >> CPython core team will work together in order to address the needs of >> both." >> >> If pip maintainers don't feel they are bound by PEPs, you could argue >> that you are also not bound by PEP 435, which would result in a >> rather pointless cooperation setup :-) >> >> Note that I'm not trying to say that you are actually not respecting the >> PEP process. I'm just concerned about comments like the above causing >> unnecessary heat in discussions. > > pip's UX decisions aren't likely to ever be put through the PEP process > again - the PEP 426 (and now PEP 470) model of providing functional > requirements and recommendations in the form of MUST and SHOULD statements > is a cleaner process, since they provide guidance for all clients, not just > pip, and leave the *details* of the UX to the normal pip development cycle > (so if user feedback indicates a problem with the specifics of the initial > approach, they can address that while remaining compliant with the > specification). > > Decoupling functional specifications from UX details of individual tools is > a good idea in general, this is just applying that model to pip and the PEP > process in particular. IMO, specific user interface questions are PEP relevant if they affect the way users interact with the Python ecosystem. This doesn't mean mandating specific option names, but e.g. --using-silly-long-options-that-scare-away-users does have PEP relevance. A PEP would have to address such user interface designs by defining whether or not to encourage or discourage certain uses. And, of course, pip as officially sanctioned Python installer would need to implement these requirements. > PyPI needs to be covered in more detail, however, as these PEPs also serve > as the *interface* specification for both clients and servers, and those > need concrete API definitions to work with. > > PEP 438 was the main case so far where the PEP included specific UX design > details for pip, and that's the aspect that *won't* be repeated. PEPs are not set in stone. They can be updated and replaced with new ones. That's why they are called "Python Enhancement *Proposals*" :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Wed Oct 8 20:09:11 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 08 Oct 2014 20:09:11 +0200 Subject: [Distutils] PEP470 installation security problems In-Reply-To: References: <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> <20141008114049.GU7954@merlinux.eu> <20141008121738.GV7954@merlinux.eu> <54353454.206@egenix.com> Message-ID: <54357DC7.2070203@egenix.com> On 08.10.2014 15:15, Paul Moore wrote: > On 8 October 2014 13:55, M.-A. Lemburg wrote: >> If pip decides to go with a strategy that ignores this, I think we >> have a problem. The core developers put trust into pip when allowing >> it to (effectively) get distributed with Python and making it the >> default Python packaging manager. Please use that trust with the >> appropriate care and respect. > > Just to clarify - the pip team (I hope I speak for all of us) fully > understand the implications of being the de facto standard package > manager. And we appreciate the trust placed in us by the fact that pip > is distributed with Python. But at the same time, that trust was given > on the basis that (presumably) we have a track record of doing things > right, in an area that is notoriously full of heated discussions and > conflicting opinions. So what we'd like to do is to continue handling > things in the same way as always, working with the packaging > community. > > In particular, that means that we did not align ourselves to the > CPython development model (as it is designed for a very different > community and set of problems). But we do want to adopt their good > practices where possible and appropriate. One of those is the PEP > process - but it's not entirely suitable (see the trail of PEPs from > the distribute/packaging/distutils2 era, for why). So we're trying to > get things right, and in the process we're learning - for example, the > failure of PEP 438 taught us that specifying installer behaviour too > closely in a PEP means we can't fix problems that are completely > messing up our users. But we still believe in the PEP process (anyone > who thinks otherwise hasn't noticed the amount of effort Donald, in > particular, is putting into all the PEPs in progress). It doesn't mean > that it can be treated as a way of forcing us not to do what we think > is right for the pip user base, though. Thanks for your clarification, Paul. I just want to remind everyone that PEPs can be augments and mistakes can be fixed by superseding one PEP with another. It's a well working process, one that is accepted in Python land and in line with the core development process. Since pip now is part of the Python stdlib (even though not bound by its release process), and the pip user base is identical with the CPython user base, the PEP process also applies to pip. That's the consequence of playing the role of an officially sanctioned part of the ecosystem and comes as part of the responsibility resulting from PEP 435. So far this has worked out well, which is why I'm surprised by some statements in this discussion. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Wed Oct 8 20:35:21 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 08 Oct 2014 20:35:21 +0200 Subject: [Distutils] PEP470 installation security problems In-Reply-To: <21FCC295-4B82-47E6-9997-83478E695D24@stufft.io> References: <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> <20141008114049.GU7954@merlinux.eu> <20141008121738.GV7954@merlinux.eu> <54353454.206@egenix.com> <67501766-E761-4E11-859F-93233B8AF9B8@stufft.io> <54353EC9.1040608@egenix.com> <21FCC295-4B82-47E6-9997-83478E695D24@stufft.io> Message-ID: <543583E9.4070804@egenix.com> On 08.10.2014 16:04, Donald Stufft wrote: > >> I'd also like to request that you take Holger's concerns more >> seriously, perhaps add him as PEP author and let him participate >> in clarifying it (if he still feels like investing time in this). > > I take all concerns and feedback seriously else I wouldn?t spend the many > hours I?ve spent just this morning responding to them. I don?t grok what > Holger?s actual concern is so it?s hard to turn those concerns into anything > actionable I can actually do on the PEP. Holger has made his points very clear in his emails. If you don't follow/grok his reasoning it may indeed be better to have him edit the PEP to add his improvements/changes. I share his view that it is not necessary to break existing setups to add multi-index support. This can be implemented as simple extension to what we already have: """ Simply add the possibility for authors to register external indexes, have pip, setuptools, et al. crawl these in addition to what's up on the PyPI package page (using the logic that has existed in these tools for years) and then let the author decide whether they want to remove existing downloads from PyPI or not. This allows for older installations to continue working, while also (optionally) supporting a setup which does not use PyPI for hosting at all. """ BTW: For eGenix we've chosen to use a different approach, one that is based on a Python web installer. I gave a talk about this at PyCon UK, in case you're interested: https://downloads.egenix.com/python/PyCon-UK-2014-Python-Web-Installer-Talk.pdf (talk video here: http://www.egenix.com/library/presentations/PyCon-UK-2014-Python-Web-Installer/) This solves the issues with the pip user experience for our packages, solves the download selection issues for the binaries, works with all Python versions we support and assures that the downloads are safe. It's still work in progress, but already quite usable. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From p.f.moore at gmail.com Wed Oct 8 20:55:56 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 8 Oct 2014 19:55:56 +0100 Subject: [Distutils] PEP470 installation security problems In-Reply-To: <54357DC7.2070203@egenix.com> References: <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> <20141008114049.GU7954@merlinux.eu> <20141008121738.GV7954@merlinux.eu> <54353454.206@egenix.com> <54357DC7.2070203@egenix.com> Message-ID: On 8 October 2014 19:09, M.-A. Lemburg wrote: > Thanks for your clarification, Paul. In the interest of making sure everyone is understanding each other, I'm going to follow up on this. I think there are some perceptions that differ slightly, and some concerns that people have, that make this a sensitive subject. I hope that by being open, I don't misword something and cause offence or concern. Like you, I'm aware that the process with pip has worked out well so far, and I don't want to disrupt that. > I just want to remind everyone that PEPs can be augments and mistakes > can be fixed by superseding one PEP with another. It's a well > working process, one that is accepted in Python land and in line > with the core development process. I think the intention with PEP 470 is precisely that, to supersede PEP 438. It's unfortunate that PEP 438 didn't work out so well in practice, but we've learned some lessons, and hopefully we're getting better at how we handle this. Specifically, I think that on the client side, PEP 438 defined too many implementation details and didn't work in terms of functional requirements. There's a tension here in that PEPs have to speak in terms of "installers" and not target pip specifically, as it's important to us that pip competes on an equal footing with other installers, and we don't act as if we have a privileged position. It's also important that the relevant PEPs are actually PEPs about *PyPI* changes. The installer details are simply advice to installers on how to adapt to those changes. But the pip team takes that a stage further and tries to ensure that we "eat our own dogfood" and implement the advice that is included in the PEPs. It's relevant in that context that the most popular "other installer" (easy_install) does not always follow the recommendations in the PEPs, so all of the negative user feedback to UI changes gets directed at pip rather than at "the PEP" or some other general target. > Since pip now is part of the Python stdlib (even though not bound > by its release process), and the pip user base is identical with > the CPython user base, the PEP process also applies to pip. My perception is somewhat different (although the practical results are similar, so this is more for context than anything else). It's important to me that pip is *not* part of the stdlib, as the release cycles of the stdlib are too slow for pip. What is part of the stdlib is *ensurepip*, which is a mechanism to install pip into a Python installation. When the subject of pip being included with Python came up, the key concern from the pip side of things was that we are still in a process of dynamic change (wheel support is still under development and improvement, for example) and the constraints of core Python / stdlib would stifle that development. Specifically, the "Policies and Governance" section of PEP 453 explicitly notes that "the bootstrapped software" (i.e. pip) will not come under CPython policies, while still noting the need for co-operation. Once again, it's worth noting that PEPs 438 and 470 are focused on *PyPI*, rather than on pip. Being a good example of an installer by following the recommendations in the PEPs for "installation programs" is part of the pip team's responsibility to work together with the CPython team, not a requirement of PEP 453. I'm completely aware, by the way, that it's pretty naive to speak of pip as merely one of many "installation programs" when in fact there's very few real alternatives. We do so specifically to keep ourselves honest... > That's the consequence of playing the role of an officially > sanctioned part of the ecosystem and comes as part of the > responsibility resulting from PEP 435. I hope the above explains why I see pip's responsibilities slightly differently. In practical terms, I think it's unlikely that our differing perspectives will result in any real differences. > So far this has worked out well, which is why I'm surprised > by some statements in this discussion. There has been a fair bit of frustration in this discussion, and that has resulted in some statements that have maybe been a little more black and white than they should have been. But frankly I think everyone has been working really hard to try to understand each others' perspectives, and I hope that will continue. That's basically why I wrote this note - I'm hoping that it's a bit clearer now why the pip team are sensitive to strong statements like "the PEP process applies to pip" which are oversimplifications of a rather complex and still evolving relationship between the core CPython and PyPA teams. Paul From p.f.moore at gmail.com Wed Oct 8 21:09:34 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 8 Oct 2014 20:09:34 +0100 Subject: [Distutils] PEP470 installation security problems In-Reply-To: <543583E9.4070804@egenix.com> References: <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> <20141008114049.GU7954@merlinux.eu> <20141008121738.GV7954@merlinux.eu> <54353454.206@egenix.com> <67501766-E761-4E11-859F-93233B8AF9B8@stufft.io> <54353EC9.1040608@egenix.com> <21FCC295-4B82-47E6-9997-83478E695D24@stufft.io> <543583E9.4070804@egenix.com> Message-ID: On 8 October 2014 19:35, M.-A. Lemburg wrote: > On 08.10.2014 16:04, Donald Stufft wrote: >> >>> I'd also like to request that you take Holger's concerns more >>> seriously, perhaps add him as PEP author and let him participate >>> in clarifying it (if he still feels like investing time in this). >> >> I take all concerns and feedback seriously else I wouldn?t spend the many >> hours I?ve spent just this morning responding to them. I don?t grok what >> Holger?s actual concern is so it?s hard to turn those concerns into anything >> actionable I can actually do on the PEP. > > Holger has made his points very clear in his emails. > > If you don't follow/grok his reasoning it may indeed be better to > have him edit the PEP to add his improvements/changes. > > I share his view that it is not necessary to break existing > setups to add multi-index support. This can be implemented as > simple extension to what we already have: > > """ > Simply add the possibility for authors to register external indexes, > have pip, setuptools, et al. crawl these in addition to what's > up on the PyPI package page (using the logic that has existed in > these tools for years) and then let the author decide whether they > want to remove existing downloads from PyPI or not. > > This allows for older installations to continue working, while > also (optionally) supporting a setup which does not use PyPI for > hosting at all. > """ OK, thanks for restating/clarifying. This was buried somewhat in the extended debate about security concerns and the implications of using multi-index support in contexts not relevant to the PEP. For what it's worth, I am -1 on this suggested addition. My reasons are: 1. The additional complexity of crawling extra indexes like this makes it harder to write new tools, or adhoc utilities (I know, I've tried :-)). 2. As the proposal stands, I don't see any way that I as a user can exercise any choice. Without inspecting the PyPI index page, I cannot know if "pip install foo" will access another website, which may be contrary to my company policy. 3. What if I want to prohibit that external access (maybe by adding a local index containing a verified copy of the package)? Unless I disable PyPI access, which may not be acceptable for other reasons, the crawl will still happen. 3. Dependency handling makes this even worse. What if a package I require, fully hosted on PyPI, depends on another one that is hosted elsewhere. How would I know? That'll do for now. Maybe Holger has proposals to address these concerns, which would be fine. It's much easier to address specifics rather than debating general, badly understood points. Once again, thanks for picking out the key point here. Paul From donald at stufft.io Wed Oct 8 21:05:18 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 8 Oct 2014 15:05:18 -0400 Subject: [Distutils] PEP470 installation security problems In-Reply-To: <543583E9.4070804@egenix.com> References: <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> <20141008114049.GU7954@merlinux.eu> <20141008121738.GV7954@merlinux.eu> <54353454.206@egenix.com> <67501766-E761-4E11-859F-93233B8AF9B8@stufft.io> <54353EC9.1040608@egenix.com> <21FCC295-4B82-47E6-9997-83478E695D24@stufft.io> <543583E9.4070804@egenix.com> Message-ID: > On Oct 8, 2014, at 2:35 PM, M.-A. Lemburg wrote: > > On 08.10.2014 16:04, Donald Stufft wrote: >> >>> I'd also like to request that you take Holger's concerns more >>> seriously, perhaps add him as PEP author and let him participate >>> in clarifying it (if he still feels like investing time in this). >> >> I take all concerns and feedback seriously else I wouldn?t spend the many >> hours I?ve spent just this morning responding to them. I don?t grok what >> Holger?s actual concern is so it?s hard to turn those concerns into anything >> actionable I can actually do on the PEP. > > Holger has made his points very clear in his emails. > > If you don't follow/grok his reasoning it may indeed be better to > have him edit the PEP to add his improvements/changes. > > I share his view that it is not necessary to break existing > setups to add multi-index support. This can be implemented as > simple extension to what we already have: > > """ > Simply add the possibility for authors to register external indexes, > have pip, setuptools, et al. crawl these in addition to what's > up on the PyPI package page (using the logic that has existed in > these tools for years) and then let the author decide whether they > want to remove existing downloads from PyPI or not. > > This allows for older installations to continue working, while > also (optionally) supporting a setup which does not use PyPI for > hosting at all. > ?"" His backwards compatibility point I understood completely and I responded that I believe that maintaining backwards compatibility for a minority of projects, where that backwards compatibility is almost entirely unsafe, is not more important than making ``pip install`` safe and it is actively preventing us making the pip repository code better. The PEP reflects that view, Holger and you may not agree with it and that's fine, I'm not going to compromise that with maintaining compat for a tiny fraction of people/projects. The thing I don't understand is Holger's worry about using the multi repository support for private projects. The PEP is entirely about using an existing feature for public projects. Private projects are mentioned a grand total of once in the entire PEP and that's just saying that the ability to specify other repositories is also useful for... and then lists a couple items, one of which is internal company repositories. So I'm not sure what I'm supposed to do with Holger's concern about --extra-index-url which don't apply to the PEP at all as far as I can tell which is where I'm looking for some clarification. How does Holger want me to address the use of this feature in a mostly unrelated to the PEP fashion in this PEP? > > BTW: For eGenix we've chosen to use a different approach, one > that is based on a Python web installer. I gave a talk about this at > PyCon UK, in case you're interested: > https://downloads.egenix.com/python/PyCon-UK-2014-Python-Web-Installer-Talk.pdf > (talk video here: > http://www.egenix.com/library/presentations/PyCon-UK-2014-Python-Web-Installer/) > This solves the issues with the pip user experience for our packages, > solves the download selection issues for the binaries, works with > all Python versions we support and assures that the downloads > are safe. It's still work in progress, but already quite usable. > I?ve been pointed at your web installer and poked at it a little bit. I don?t have any specific points about it since I only really skimmed it, hopefully you?re doing all that needs to be done to secure the downloads, but otherwise I?m glad you found something that gives you more control over the process and that still works well with the newer policies of things. Maybe it?d make sense to also explicitly mention this as an additional option? Is there a tool you?re using to manage all this or is it all one-off and specific to eGenix? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From donald at stufft.io Wed Oct 8 21:14:28 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 8 Oct 2014 15:14:28 -0400 Subject: [Distutils] PEP470 installation security problems In-Reply-To: References: <03ABBD5F-0162-4ED8-B9C8-2C50BF3D6257@stufft.io> <20141008071743.GO7954@merlinux.eu> <27C21835-3FD8-4A52-B508-9C71B7E78535@stufft.io> <20141008084456.GP7954@merlinux.eu> <20141008105736.GS7954@merlinux.eu> <20141008114049.GU7954@merlinux.eu> <20141008121738.GV7954@merlinux.eu> <54353454.206@egenix.com> <54357DC7.2070203@egenix.com> Message-ID: <8044DC74-3DCB-4263-B784-E1DCEB452E4A@stufft.io> > On Oct 8, 2014, at 2:55 PM, Paul Moore wrote: > > There's a tension here in that > PEPs have to speak in terms of "installers" and not target pip > specifically, as it's important to us that pip competes on an equal > footing with other installers, and we don't act as if we have a > privileged position. This in particular is something I feel fairly strongly about. I do *not* want another situation where everyone has to use X blessed thing (e.g. distutils) or they have to go to terrible lengths (monkey patching etc) and deal with trying to work around the expectation that there would be only one tool (ez_setup.py imports in setup.py). That's why a big focus of mine has been on standardizing things via the PEP process and making them formats and APIs not implementations. I want someone to be able to come along and say "I really "hate/think I can do better than" this pip thing and they have well documented standards and the ability to just ignore pip all together. Now I also hope that pip is good enough that people don't want to do that, but I want it to be entirely possible if they do. That's why you'll see my PEPs reference what high level features an "installer" should implement but you'll not find much detail beyond that. I do try to include examples and to call out what behaviors are currently in play though just so that there's some example and "prior art" sort of information in the PEP. pip has a privileged position amongst installers (and possible future ones!) and we don't want that privelege to be exasperated by making all of our PEPs pip specific. We have a further privelege in that I'm both a pip core developer and a PyPI admin/developer so it would be fairly easy for me to add special APIs just for pip, but again I feel strongly about pip not being special in these situations so we restrict ourselves. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From holger at merlinux.eu Thu Oct 9 18:41:39 2014 From: holger at merlinux.eu (holger krekel) Date: Thu, 9 Oct 2014 16:41:39 +0000 Subject: [Distutils] some questions about PEP470 Message-ID: <20141009164139.GE7954@merlinux.eu> Hi Donald, Nick, to change the somewhat unsuccessfull way how we were conversing about PEP470 so far i'd like to kindly ask you a few questions related to the PEP. This is to check if i am maybe barking up the wrong tree and also to enlarge the common ground/understanding that we are discussing on. Please take your time, i'd appreciate if you give a joint answer rather than a quick one. thanks, holger Numbers of users affected --------------------------------- Do i see it right that the PEP470 changes would mean about 6-7 thousand users (per day) need to change their installation options to use "--extra-index-url"? If not, how many? Is there a monthly figure? And that the affected users can only do that if the respective maintainers of the projects offer an external index (or re-upload to PyPI)? Do i see it right that up to a 1000 maintainers need to act and offer an external index if they want to keep their projects properly installable? What can users of (older or current) easy_install versions do if they want to install a project with external links in a post PEP470 world? How many people currently use easy_install/not-pip (numbers, not only percentages)? I've understood you made these two statements during the discussion: - PEP438 caused bad UI for dealing with pypi-external links -- many people are confused by it and we thus need to fix it. - PEP470 breaking backward compatibility for pypi-external links is not a big deal because it affects only a tiny fraction of the users. Could you choose which one of them you consider is true? Recommendation of "--extra-index-url" -------------------------------------- In your mind and forgetting about PEP470, in what situations exactly is "pip install --extra-index-url" a safe option for users? Interpretation of external link usage -------------------------------------------- In the main rationale you say: "While a large number of projects did ultimately decide to upload to PyPI, some of them did so only because the UX around what PEP 438 was so bad that they felt forced to do so." Could you provide some tractable background (not just your strong opinion) for this interpretation? Why can it not be that people nowadays just prefer to upload to PyPI without even considering alternative options? From donald at stufft.io Thu Oct 9 19:40:05 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 9 Oct 2014 13:40:05 -0400 Subject: [Distutils] some questions about PEP470 In-Reply-To: <20141009164139.GE7954@merlinux.eu> References: <20141009164139.GE7954@merlinux.eu> Message-ID: <89513014-4E0E-422D-AD12-95730EED60A8@stufft.io> > On Oct 9, 2014, at 12:41 PM, holger krekel wrote: > > Hi Donald, Nick, > > to change the somewhat unsuccessfull way how we were conversing about PEP470 > so far i'd like to kindly ask you a few questions related to the PEP. > This is to check if i am maybe barking up the wrong tree and also to > enlarge the common ground/understanding that we are discussing on. > > Please take your time, i'd appreciate if you give a joint > answer rather than a quick one. > > thanks, > holger > > > Numbers of users affected > --------------------------------- > > Do i see it right that the PEP470 changes would mean about 6-7 thousand > users (per day) need to change their installation options to use > "--extra-index-url"? If not, how many? Is there a monthly figure? It?s impossible to couch this in terms of ?users? because we have no way of correlating what we see on the PyPI side with users. On the single day I selected to look at the logs (which was more or less the day before the day I was compounding numbers) there were 6.6k total unique IP addresses that hit a /simple/ page which belonged to one of the affected projects. Beyond knowing how many IP addresses it?s difficult to determine how that correlates into users, that could be a single user with 6.6k different EC2 machines, or it could be 6.6k individual users (or even more than that if there is a transparent proxy at play). In all likelihood it is not a single user and it is not 6.6k users but somewhere in between. Important to point out that this number also includes people spinning up bandersnatch mirrors, devpi mirrors, or any other automated fetching of the /simple/ page for reasons other than ?I?d like to install this project?. > > And that the affected users can only do that if the respective > maintainers of the projects offer an external index (or re-upload to PyPI)? No and Yes. Wherever pip/easy_install are currently finding the download from can serve as the external index. This likely won?t be the most efficient repository since often times these are regular web pages which have other content and the like but it won?t be any worse than it is currently. For instance you can take a look at https://bpaste.net/show/5a83985ad2e6 to see using the current page as a find-links repository with pip. > > Do i see it right that up to a 1000 maintainers need to act and offer an > external index if they want to keep their projects properly installable? If their project is already installable, then they already have something which is usable as either a simple or a find-links repository. The only action required on their part is if they want the discovery affordances in this PEP they would need to tell PyPI that. > > What can users of (older or current) easy_install versions do if they > want to install a project with external links in a post PEP470 world? They can use ?find-links. > > How many people currently use easy_install/not-pip (numbers, not only > percentages)? Later I can create a similar graph with absolute numbers, but here?s a graph showing percentages for all downloads from PyPI. This does not include people strictly downloading external links since this is looking specifically at the access logs for /packages/*, but it is likely to be fairly representative. http://d.stufft.io/image/0r0X3Y2M100p > > I've understood you made these two statements during the discussion: > > - PEP438 caused bad UI for dealing with pypi-external links -- > many people are confused by it and we thus need to fix it. > > - PEP470 breaking backward compatibility for pypi-external links is > not a big deal because it affects only a tiny fraction of the users. > > Could you choose which one of them you consider is true? I consider them both to be true. The PEP 438 UX is confusing, out of the people who have had to use it I have seem a fairly high percentage of those completely confused by it. It, especially right when pip 1.5 was released, was one of our most reported issues. The total number of people who need to use it has gone down over time, however I still believe that percentage wise most people who need to use it are confused by it. I do not believe that PEP 470 breaking backwards compatability for pypi-external links to be a terrible burden because it only affects a small percentage of the total users of PyPI. I think perhaps the reason you think both of them can?t be true is you?re assuming that I?m talking about percentages of the same total population? > > > Recommendation of "--extra-index-url" > -------------------------------------- > > In your mind and forgetting about PEP470, in what situations exactly is > "pip install --extra-index-url" a safe option for users? The answer to this isn?t really related to ?extra-index-url, ``pip install foo`` is ?safe? (given the threat model we operate under) if, and only if, you trust the operators of all of the repositories you have configured (by default, via ?index-url, via ?extra-index-url, via ?find-links, and via ?process-dependency-links), to give you the correct files for ?foo?. How the repositories have come to be configured isn?t particularly meaningful. > > > Interpretation of external link usage > -------------------------------------------- > > In the main rationale you say: > > "While a large number of projects did ultimately decide to upload to > PyPI, some of them did so only because the UX around what PEP 438 was so > bad that they felt forced to do so." > > Could you provide some tractable background (not just your strong opinion) > for this interpretation? Why can it not be that people nowadays just > prefer to upload to PyPI without even considering alternative options? > Well Stefan had voiced that complaint last time that he felt we were trying to force him to upload to PyPI by making the UX so bad. I?ve had a few other people say similar things to me in private. > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From holger at merlinux.eu Sat Oct 11 08:31:48 2014 From: holger at merlinux.eu (holger krekel) Date: Sat, 11 Oct 2014 06:31:48 +0000 Subject: [Distutils] some questions about PEP470 In-Reply-To: <89513014-4E0E-422D-AD12-95730EED60A8@stufft.io> References: <20141009164139.GE7954@merlinux.eu> <89513014-4E0E-422D-AD12-95730EED60A8@stufft.io> Message-ID: <20141011063148.GN7954@merlinux.eu> Hi Donald, many thanks for answering. A few follow up questions inline. On Thu, Oct 09, 2014 at 13:40 -0400, Donald Stufft wrote: > > On Oct 9, 2014, at 12:41 PM, holger krekel wrote: > > > Numbers of users affected > > --------------------------------- > > > > Do i see it right that the PEP470 changes would mean about 6-7 thousand > > users (per day) need to change their installation options to use > > "--extra-index-url"? If not, how many? Is there a monthly figure? > > > It?s impossible to couch this in terms of ?users? because we have no way > of correlating what we see on the PyPI side with users. On the single day > I selected to look at the logs (which was more or less the day before the > day I was compounding numbers) there were 6.6k total unique IP addresses > that hit a /simple/ page which belonged to one of the affected projects. > > Beyond knowing how many IP addresses it?s difficult to determine how that > correlates into users, that could be a single user with 6.6k different EC2 > machines, or it could be 6.6k individual users (or even more than that if > there is a transparent proxy at play). In all likelihood it is not a single > user and it is not 6.6k users but somewhere in between. > > Important to point out that this number also includes people spinning up > bandersnatch mirrors, devpi mirrors, or any other automated fetching of > the /simple/ page for reasons other than ?I?d like to install this project?. I understand it's hard to get to somewhat sensible numbers and the number of unique IPs is probably only an upper bound. devpi and bandersnatch make even that fuzzy because more than one user may be behind each such instance. Anyway, can you provide a monthly number of unique IP addresses on the simple pages on projects with external links? > > And that the affected users can only do that if the respective > > maintainers of the projects offer an external index (or re-upload to PyPI)? > > No and Yes. > > Wherever pip/easy_install are currently finding the download from can serve > as the external index. This likely won?t be the most efficient repository > since often times these are regular web pages which have other content and > the like but it won?t be any worse than it is currently. For instance you > can take a look at https://bpaste.net/show/5a83985ad2e6 to see using the > current page as a find-links repository with pip. How can affected users discover they need to use this particular option and URL if they use today's pip/easy_install versions and a post PEP470 PyPI? > > Do i see it right that up to a 1000 maintainers need to act and offer an > > external index if they want to keep their projects properly installable? > > If their project is already installable, then they already have something > which is usable as either a simple or a find-links repository. The only > action required on their part is if they want the discovery affordances > in this PEP they would need to tell PyPI that. Is this true also for the (small but still) set of maintainers who registered external links with checksums? If maintainers don't act, will using a post PEP470 released pip help the users in any way? > > I've understood you made these two statements during the discussion: > > > > - PEP438 caused bad UI for dealing with pypi-external links -- > > many people are confused by it and we thus need to fix it. > > > > - PEP470 breaking backward compatibility for pypi-external links is > > not a big deal because it affects only a tiny fraction of the users. > > > > Could you choose which one of them you consider is true? > > I consider them both to be true. > > The PEP 438 UX is confusing, out of the people who have had to use it I > have seem a fairly high percentage of those completely confused by it. It, > especially right when pip 1.5 was released, was one of our most reported > issues. The total number of people who need to use it has gone down over > time, however I still believe that percentage wise most people who need to > use it are confused by it. > > I do not believe that PEP 470 breaking backwards compatability for pypi-external > links to be a terrible burden because it only affects a small percentage of the > total users of PyPI. > > I think perhaps the reason you think both of them can?t be true is you?re > assuming that I?m talking about percentages of the same total population? Yes, i was assuming that for both statements the same basis group was used. So i understand know you are saying overall very few people depend on external links but out of those who do, many are confused and annoyed about how it works. Will the people who suffered from the current external linking options be the same ones who could be affected by backward compatibility issues (i.e. commands which now work can fail with a post-PEP470 PyPI server)? personal side question: do i remember correctly that when we discussed PEP438 you pushed for the current set of behaviours wrt to external links while i tried to keep it simpler because you put higher priority on protection against MITM attacks? > > Recommendation of "--extra-index-url" > > -------------------------------------- > > > > In your mind and forgetting about PEP470, in what situations exactly is > > "pip install --extra-index-url" a safe option for users? > > > The answer to this isn?t really related to ?extra-index-url, ``pip install foo`` > is ?safe? (given the threat model we operate under) if, and only if, you trust > the operators of all of the repositories you have configured (by default, via > ?index-url, via ?extra-index-url, via ?find-links, and via ?process-dependency-links), > to give you the correct files for ?foo?. How the repositories have come to be > configured isn?t particularly meaningful. I understand that as a fairly generic security statement. But I was trying to rather ask about use cases and scenarios where precisely the --extra-index-url option is useful and to be recommended. I'd be grateful if Nick or you could still describe use cases, especially outside PEP470 external links context (the option existed before so i presume there must be some use cases). > > Interpretation of external link usage > > -------------------------------------------- > > > > In the main rationale you say: > > > > "While a large number of projects did ultimately decide to upload to > > PyPI, some of them did so only because the UX around what PEP 438 was so > > bad that they felt forced to do so." > > > > Could you provide some tractable background (not just your strong opinion) > > for this interpretation? Why can it not be that people nowadays just > > prefer to upload to PyPI without even considering alternative options? > > > > Well Stefan had voiced that complaint last time that he felt we were trying > to force him to upload to PyPI by making the UX so bad. I?ve had a few other > people say similar things to me in private. I can sympathize. In fact, I think we didn't deliver the upload tools that we outlined with PEP438, particularly registration of externally verified links. My bad as well. best, holger From donald at stufft.io Sat Oct 11 20:27:16 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 11 Oct 2014 14:27:16 -0400 Subject: [Distutils] some questions about PEP470 In-Reply-To: <20141011063148.GN7954@merlinux.eu> References: <20141009164139.GE7954@merlinux.eu> <89513014-4E0E-422D-AD12-95730EED60A8@stufft.io> <20141011063148.GN7954@merlinux.eu> Message-ID: <264E2355-E619-476E-992B-8D11BBBFB217@stufft.io> > On Oct 11, 2014, at 2:31 AM, holger krekel wrote: > > Hi Donald, > > many thanks for answering. A few follow up questions inline. > > On Thu, Oct 09, 2014 at 13:40 -0400, Donald Stufft wrote: >>> On Oct 9, 2014, at 12:41 PM, holger krekel wrote: >> >>> Numbers of users affected >>> --------------------------------- >>> >>> Do i see it right that the PEP470 changes would mean about 6-7 thousand >>> users (per day) need to change their installation options to use >>> "--extra-index-url"? If not, how many? Is there a monthly figure? >> >> >> It?s impossible to couch this in terms of ?users? because we have no way >> of correlating what we see on the PyPI side with users. On the single day >> I selected to look at the logs (which was more or less the day before the >> day I was compounding numbers) there were 6.6k total unique IP addresses >> that hit a /simple/ page which belonged to one of the affected projects. >> >> Beyond knowing how many IP addresses it?s difficult to determine how that >> correlates into users, that could be a single user with 6.6k different EC2 >> machines, or it could be 6.6k individual users (or even more than that if >> there is a transparent proxy at play). In all likelihood it is not a single >> user and it is not 6.6k users but somewhere in between. >> >> Important to point out that this number also includes people spinning up >> bandersnatch mirrors, devpi mirrors, or any other automated fetching of >> the /simple/ page for reasons other than ?I?d like to install this project?. > > I understand it's hard to get to somewhat sensible numbers and the > number of unique IPs is probably only an upper bound. devpi and bandersnatch > make even that fuzzy because more than one user may be behind each such > instance. Anyway, can you provide a monthly number of unique IP addresses on > the simple pages on projects with external links? I can do that, it?ll take a little while because the scripts I have to do it aren?t really designed to handle more than a single file and the files are quite large once decompressed (30 days is roughly 120GB of log files). > >>> And that the affected users can only do that if the respective >>> maintainers of the projects offer an external index (or re-upload to PyPI)? >> >> No and Yes. >> >> Wherever pip/easy_install are currently finding the download from can serve >> as the external index. This likely won?t be the most efficient repository >> since often times these are regular web pages which have other content and >> the like but it won?t be any worse than it is currently. For instance you >> can take a look at https://bpaste.net/show/5a83985ad2e6 to see using the >> current page as a find-links repository with pip. > > How can affected users discover they need to use this particular option > and URL if they use today's pip/easy_install versions and a post PEP470 PyPI? I plan to put the external repositories (and the commands needed to use them) in the UI for PyPI. I suppose I should put that in the PEP as well, I was more focused on defining the API differences and the changes. > >>> Do i see it right that up to a 1000 maintainers need to act and offer an >>> external index if they want to keep their projects properly installable? >> >> If their project is already installable, then they already have something >> which is usable as either a simple or a find-links repository. The only >> action required on their part is if they want the discovery affordances >> in this PEP they would need to tell PyPI that. > > Is this true also for the (small but still) set of maintainers who > registered external links with checksums? A find-link repository can be as simple as a direct link to a file yes, it?s not the best idea since you can?t ever change the location of that file without breaking it for people but it can be done. > > If maintainers don't act, will using a post PEP470 released pip help > the users in any way? Absolutely! A common complaint that I?ve seen from users is that when faced with network errors pip?s output is extremely unhelpful. For instance here?s an issue about how TLS errors are presented - https://github.com/pypa/pip/issues/1511. The primary reason for this is that because we don?t know if a particular link *should* be up, or if it?s expected to be down (xfail, in testing lingo) we have to assume that it should be down. Sometimes there can be many of these links so to avoid having lots of extraneous ?error? looking messages in the output we just silently hide them. However if none of the links can be found they?ll just get a confusing ?Sorry can?t find anything to install for that?. Worse yet, if we can find some things but not everything then we might install an older insecure version. This could actually be used to prevent people from getting a security update for a particular project if they?ve switched from hosting on PyPI to hosting externally and a MITM attacker is blocking their attempts to reach the external host. A post PEP 470 pip will be able to treat all URLs it fetches as mandatory, so it can raise proper error messages when it can?t locate an URL to communicate that to the end user and ideally make things far less confusing. > >>> I've understood you made these two statements during the discussion: >>> >>> - PEP438 caused bad UI for dealing with pypi-external links -- >>> many people are confused by it and we thus need to fix it. >>> >>> - PEP470 breaking backward compatibility for pypi-external links is >>> not a big deal because it affects only a tiny fraction of the users. >>> >>> Could you choose which one of them you consider is true? >> >> I consider them both to be true. >> >> The PEP 438 UX is confusing, out of the people who have had to use it I >> have seem a fairly high percentage of those completely confused by it. It, >> especially right when pip 1.5 was released, was one of our most reported >> issues. The total number of people who need to use it has gone down over >> time, however I still believe that percentage wise most people who need to >> use it are confused by it. >> >> I do not believe that PEP 470 breaking backwards compatability for pypi-external >> links to be a terrible burden because it only affects a small percentage of the >> total users of PyPI. >> >> I think perhaps the reason you think both of them can?t be true is you?re >> assuming that I?m talking about percentages of the same total population? > > Yes, i was assuming that for both statements the same basis group was used. > So i understand know you are saying overall very few people depend on > external links but out of those who do, many are confused and annoyed > about how it works. Yes exactly. > > Will the people who suffered from the current external linking options > be the same ones who could be affected by backward compatibility issues > (i.e. commands which now work can fail with a post-PEP470 PyPI server)? Yes-ish. A lot of those people switched away from external hosting all together or chose to rely on different projects that didn?t need that. It would likely be a subset of those users. > > personal side question: do i remember correctly that when we discussed > PEP438 you pushed for the current set of behaviours wrt to external > links while i tried to keep it simpler because you put higher priority > on protection against MITM attacks? I have very little recollection of the discussion around PEP 438, I have a bad memory for details like that. It?s entirely possible and sounds like me to be security focused. > >>> Recommendation of "--extra-index-url" >>> -------------------------------------- >>> >>> In your mind and forgetting about PEP470, in what situations exactly is >>> "pip install --extra-index-url" a safe option for users? >> >> >> The answer to this isn?t really related to ?extra-index-url, ``pip install foo`` >> is ?safe? (given the threat model we operate under) if, and only if, you trust >> the operators of all of the repositories you have configured (by default, via >> ?index-url, via ?extra-index-url, via ?find-links, and via ?process-dependency-links), >> to give you the correct files for ?foo?. How the repositories have come to be >> configured isn?t particularly meaningful. > > I understand that as a fairly generic security statement. But I was trying to > rather ask about use cases and scenarios where precisely the > --extra-index-url option is useful and to be recommended. > I'd be grateful if Nick or you could still describe use cases, > especially outside PEP470 external links context (the option existed > before so i presume there must be some use cases). I?m not sure exactly what you want here. It?s hard to come up with an exhaustive list of scenarios where it?s useful. One I?ve used in the past was: -i https://index.example.com/base/ ?extra-index-url https://index.example.com/project-foo-overrides/ > >>> Interpretation of external link usage >>> -------------------------------------------- >>> >>> In the main rationale you say: >>> >>> "While a large number of projects did ultimately decide to upload to >>> PyPI, some of them did so only because the UX around what PEP 438 was so >>> bad that they felt forced to do so." >>> >>> Could you provide some tractable background (not just your strong opinion) >>> for this interpretation? Why can it not be that people nowadays just >>> prefer to upload to PyPI without even considering alternative options? >>> >> >> Well Stefan had voiced that complaint last time that he felt we were trying >> to force him to upload to PyPI by making the UX so bad. I?ve had a few other >> people say similar things to me in private. > > I can sympathize. In fact, I think we didn't deliver the upload tools > that we outlined with PEP438, particularly registration of externally > verified links. My bad as well. > > best, > holger --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From donald at stufft.io Sat Oct 11 20:29:51 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 11 Oct 2014 14:29:51 -0400 Subject: [Distutils] some questions about PEP470 In-Reply-To: <264E2355-E619-476E-992B-8D11BBBFB217@stufft.io> References: <20141009164139.GE7954@merlinux.eu> <89513014-4E0E-422D-AD12-95730EED60A8@stufft.io> <20141011063148.GN7954@merlinux.eu> <264E2355-E619-476E-992B-8D11BBBFB217@stufft.io> Message-ID: <12C9C976-9B22-4D29-9CD9-C9CC66576D9D@stufft.io> > On Oct 11, 2014, at 2:27 PM, Donald Stufft wrote: >> >>>> And that the affected users can only do that if the respective >>>> maintainers of the projects offer an external index (or re-upload to PyPI)? >>> >>> No and Yes. >>> >>> Wherever pip/easy_install are currently finding the download from can serve >>> as the external index. This likely won?t be the most efficient repository >>> since often times these are regular web pages which have other content and >>> the like but it won?t be any worse than it is currently. For instance you >>> can take a look at https://bpaste.net/show/5a83985ad2e6 to see using the >>> current page as a find-links repository with pip. >> >> How can affected users discover they need to use this particular option >> and URL if they use today's pip/easy_install versions and a post PEP470 PyPI? > > I plan to put the external repositories (and the commands needed to use them) > in the UI for PyPI. I suppose I should put that in the PEP as well, I was more > focused on defining the API differences and the changes. I forgot to mention, one other thing I thought about, we could move the current external links to external repositories automatically. I?m not sure if this is a good idea because on some projects that will end up with a giant list of things that a user would be suggested to install. Perhaps it?d work to scan them for installable files and only move the ones where an installable file was discovered, though that could be error prone (for example right now bytereef.org is down, so we?d not discover any files there). --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From ncoghlan at gmail.com Sun Oct 12 01:48:50 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 12 Oct 2014 09:48:50 +1000 Subject: [Distutils] some questions about PEP470 In-Reply-To: <12C9C976-9B22-4D29-9CD9-C9CC66576D9D@stufft.io> References: <20141009164139.GE7954@merlinux.eu> <89513014-4E0E-422D-AD12-95730EED60A8@stufft.io> <20141011063148.GN7954@merlinux.eu> <264E2355-E619-476E-992B-8D11BBBFB217@stufft.io> <12C9C976-9B22-4D29-9CD9-C9CC66576D9D@stufft.io> Message-ID: On 12 October 2014 04:29, Donald Stufft wrote: >> I plan to put the external repositories (and the commands needed to use them) >> in the UI for PyPI. I suppose I should put that in the PEP as well, I was more >> focused on defining the API differences and the changes. > > I forgot to mention, one other thing I thought about, we could move the current > external links to external repositories automatically. I?m not sure if this is > a good idea because on some projects that will end up with a giant list of things > that a user would be suggested to install. Perhaps it?d work to scan them for > installable files and only move the ones where an installable file was discovered, > though that could be error prone (for example right now bytereef.org is down, so > we?d not discover any files there). If I recall Holger's suggestion correctly, it was just to move the current "Download URL" links over, rather than the scraped links. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donald at stufft.io Sun Oct 12 01:49:53 2014 From: donald at stufft.io (Donald Stufft) Date: Sat, 11 Oct 2014 19:49:53 -0400 Subject: [Distutils] some questions about PEP470 In-Reply-To: References: <20141009164139.GE7954@merlinux.eu> <89513014-4E0E-422D-AD12-95730EED60A8@stufft.io> <20141011063148.GN7954@merlinux.eu> <264E2355-E619-476E-992B-8D11BBBFB217@stufft.io> <12C9C976-9B22-4D29-9CD9-C9CC66576D9D@stufft.io> Message-ID: <8F2706F7-4804-4E20-8869-48FE075C2155@stufft.io> On Oct 11, 2014, at 7:48 PM, Nick Coghlan wrote: > On 12 October 2014 04:29, Donald Stufft wrote: >>> I plan to put the external repositories (and the commands needed to use them) >>> in the UI for PyPI. I suppose I should put that in the PEP as well, I was more >>> focused on defining the API differences and the changes. >> >> I forgot to mention, one other thing I thought about, we could move the current >> external links to external repositories automatically. I?m not sure if this is >> a good idea because on some projects that will end up with a giant list of things >> that a user would be suggested to install. Perhaps it?d work to scan them for >> installable files and only move the ones where an installable file was discovered, >> though that could be error prone (for example right now bytereef.org is down, so >> we?d not discover any files there). > > If I recall Holger's suggestion correctly, it was just to move the > current "Download URL" links over, rather than the scraped links. Hmm, I thought his suggestion was to just leave the existing links alone so that the existing pip/easy_installs would continue to function as they do now? From ncoghlan at gmail.com Sun Oct 12 02:10:20 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 12 Oct 2014 10:10:20 +1000 Subject: [Distutils] some questions about PEP470 In-Reply-To: <8F2706F7-4804-4E20-8869-48FE075C2155@stufft.io> References: <20141009164139.GE7954@merlinux.eu> <89513014-4E0E-422D-AD12-95730EED60A8@stufft.io> <20141011063148.GN7954@merlinux.eu> <264E2355-E619-476E-992B-8D11BBBFB217@stufft.io> <12C9C976-9B22-4D29-9CD9-C9CC66576D9D@stufft.io> <8F2706F7-4804-4E20-8869-48FE075C2155@stufft.io> Message-ID: On 12 October 2014 09:49, Donald Stufft wrote: > > On Oct 11, 2014, at 7:48 PM, Nick Coghlan wrote: > >> On 12 October 2014 04:29, Donald Stufft wrote: >>>> I plan to put the external repositories (and the commands needed to use them) >>>> in the UI for PyPI. I suppose I should put that in the PEP as well, I was more >>>> focused on defining the API differences and the changes. >>> >>> I forgot to mention, one other thing I thought about, we could move the current >>> external links to external repositories automatically. I?m not sure if this is >>> a good idea because on some projects that will end up with a giant list of things >>> that a user would be suggested to install. Perhaps it?d work to scan them for >>> installable files and only move the ones where an installable file was discovered, >>> though that could be error prone (for example right now bytereef.org is down, so >>> we?d not discover any files there). >> >> If I recall Holger's suggestion correctly, it was just to move the >> current "Download URL" links over, rather than the scraped links. > > Hmm, I thought his suggestion was to just leave the existing links alone so that the > existing pip/easy_installs would continue to function as they do now? Sure, we could do that too. Regardless, I don't believe we should let the conversion of the silent security failures into noisy exceptions block the other usability enhancements - if decoupling the two decisions will achieve that, then let's go down that path. I think I suggested at one point that the PEP could potentially commit to making the backwards compatible changes on the server side, and then for the *in*compatible changes, only commit to *looking at the numbers again* in a few months time after folks have had a chance to experience the replacement model. Client applications would be officially granted permission to ignore the PEP 438 external hosting mechanism in favour of the new PEP 470 external index advertisements. The sequence of events from a compatibility break/user experience enhancement perspective would then be: * new model added to PyPI (no compatibility break) * old model dropped from pip (client side compatibility break, error handling improvements) * old model dropped from PyPI (conditional, based on changes in server metrics) That changes the dynamic from "the updated user experience will be better, trust us" to "if you prefer the old user experience, you can keep using the old pip, if you prefer the new user experience, upgrade". The eventual decision on when to drop the legacy model from PyPI can then by made based on a richer set of input data, that includes the level of adoption of the new version of pip that treats all missing link targets as potential errors. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From carl at oddbird.net Sun Oct 12 02:48:18 2014 From: carl at oddbird.net (Carl Meyer) Date: Sat, 11 Oct 2014 18:48:18 -0600 Subject: [Distutils] some questions about PEP470 In-Reply-To: <20141011063148.GN7954@merlinux.eu> References: <20141009164139.GE7954@merlinux.eu> <89513014-4E0E-422D-AD12-95730EED60A8@stufft.io> <20141011063148.GN7954@merlinux.eu> Message-ID: <5439CFD2.5000909@oddbird.net> Hi Holger, On 10/11/2014 12:31 AM, holger krekel wrote: > I understand that as a fairly generic security statement. But I was trying to > rather ask about use cases and scenarios where precisely the > --extra-index-url option is useful and to be recommended. > I'd be grateful if Nick or you could still describe use cases, > especially outside PEP470 external links context (the option existed > before so i presume there must be some use cases). I don't use it anymore (because these days for everything other than interactive playing around, I install only from a curated local index specifically limited to each project's dependencies using --no-index and --find-links), but I used to use it. My use case was this: generally dependencies were installed from PyPI, but occasionally I would need to patch a dependency, so I would create an sdist with a patched version number (e.g. if I patched 2.0.1, I would create an sdist for version 2.0.1.obc1, where "obc" is a tag based on my company name or the project) and add this patched sdist to my own index, which I would add to my installs with --extra-index-url. Because I used a patched version number and pinned all dependencies exactly, it didn't matter to me that both PyPI and my "extra index" were considered for installation; in fact that was convenient, since it meant I could very easily upgrade to a newer PyPI release. I never used it for "private" non-PyPI packages. Carl -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: From p.f.moore at gmail.com Sun Oct 12 09:50:59 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 12 Oct 2014 08:50:59 +0100 Subject: [Distutils] some questions about PEP470 In-Reply-To: <5439CFD2.5000909@oddbird.net> References: <20141009164139.GE7954@merlinux.eu> <89513014-4E0E-422D-AD12-95730EED60A8@stufft.io> <20141011063148.GN7954@merlinux.eu> <5439CFD2.5000909@oddbird.net> Message-ID: On 10/11/2014 12:31 AM, holger krekel wrote: > I understand that as a fairly generic security statement. But I was trying to > rather ask about use cases and scenarios where precisely the > --extra-index-url option is useful and to be recommended. > I'd be grateful if Nick or you could still describe use cases, > especially outside PEP470 external links context (the option existed > before so i presume there must be some use cases). I use it to have a local repository of wheels for projects that do not (yet) host wheels on PyPI. I want the fallback to PyPI should newer versions be released. Paul From p.f.moore at gmail.com Sun Oct 12 13:38:26 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 12 Oct 2014 12:38:26 +0100 Subject: [Distutils] Having a "less complete" configuration for a project Message-ID: I'm building a project that allows users to create files and directory structures from static data. The core of the project will be a command line API that includes a number of features that depend on 3rd party libraries. But I can see the core of the project being useful for smaller tasks, for example generating test data. So what I'd like is to have the "standard" installation "pip install myproj" build the full application, but have a way to specify (for example in a tox config) that only the core library, without the larger dependencies, be installed. Doing this the other way round is easy using extras: pip install myproj -> installs myproj and core dependencies pip install myproj[cmdline] -> installs the above, plus jinja2, click, ... But the problem with this approach is that "normal users" need to know to specify the extra. Is it possible to switch this round somehow, so that I have an "extra" that *removes* some of the dependencies? (I could have 2 projects, a core one and a cmdline one that depends on the first and adds the command-line API. I don't really like that option, though, not least because it means maintaining 2 projects rather than one, so I'm exploring whether there are other ways of achieving the objective). Any thoughts on how I could do this? Paul From ncoghlan at gmail.com Sun Oct 12 13:54:18 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 12 Oct 2014 21:54:18 +1000 Subject: [Distutils] Having a "less complete" configuration for a project In-Reply-To: References: Message-ID: On 12 October 2014 21:38, Paul Moore wrote: > I'm building a project that allows users to create files and directory > structures from static data. The core of the project will be a command > line API that includes a number of features that depend on 3rd party > libraries. But I can see the core of the project being useful for > smaller tasks, for example generating test data. So what I'd like is > to have the "standard" installation "pip install myproj" build the > full application, but have a way to specify (for example in a tox > config) that only the core library, without the larger dependencies, > be installed. > > Doing this the other way round is easy using extras: > > pip install myproj -> installs myproj and core dependencies > pip install myproj[cmdline] -> installs the above, plus jinja2, click, ... > > But the problem with this approach is that "normal users" need to know > to specify the extra. > > Is it possible to switch this round somehow, so that I have an "extra" > that *removes* some of the dependencies? > > (I could have 2 projects, a core one and a cmdline one that depends on > the first and adds the command-line API. I don't really like that > option, though, not least because it means maintaining 2 projects > rather than one, so I'm exploring whether there are other ways of > achieving the objective). > > Any thoughts on how I could do this? I don't know of any current way to do it, and even the more flexible extras notation in PEP 426 doesn't quite get you there. The closest it currently allows is "pip install myproj[-,core]". The "-" disables installing the default dependencies, but it *also* disables installing the distribution itself. It would be possible to extend that to "pip install myproj[-,:self:,core]", where ":self:" would be a new implicit extra for installing the project itself (to go along with the currently proposed implicit extras for ":meta:", ":run:", ":build:", ":test:", and ":dev:" to indicate which dependency lists to process). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Oct 12 14:04:51 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 12 Oct 2014 22:04:51 +1000 Subject: [Distutils] Having a "less complete" configuration for a project In-Reply-To: References: Message-ID: On 12 October 2014 21:54, Nick Coghlan wrote: > On 12 October 2014 21:38, Paul Moore wrote: >> Is it possible to switch this round somehow, so that I have an "extra" >> that *removes* some of the dependencies? >> >> (I could have 2 projects, a core one and a cmdline one that depends on >> the first and adds the command-line API. I don't really like that >> option, though, not least because it means maintaining 2 projects >> rather than one, so I'm exploring whether there are other ways of >> achieving the objective). >> >> Any thoughts on how I could do this? > > I don't know of any current way to do it, and even the more flexible > extras notation in PEP 426 doesn't quite get you there. > > The closest it currently allows is "pip install myproj[-,core]". The > "-" disables installing the default dependencies, but it *also* > disables installing the distribution itself. On second thoughts, I think PEP 426 does already cover this, by letting you write "pip install myproj[-:meta:,-:run:,core]". That would turn off the normal "meta_requires" and "run_requires" processing, and just install the dependencies for the "core" extra instead (which would then presumably be a subset of the normal run_requires). You'd need to specify your core dependencies in two places (once in run_requires and once for the core extra), but having setup.py as an executable format copes with that OK. > It would be possible to extend that to "pip install > myproj[-,:self:,core]", where ":self:" would be a new implicit extra > for installing the project itself (to go along with the currently > proposed implicit extras for ":meta:", ":run:", ":build:", ":test:", > and ":dev:" to indicate which dependency lists to process). That said, I still quite like this idea of the implicit ":self:" extra. With that included, I'd drop the ability to have a standalone "-" entirely. Instead, the spelling would be "pip install myproj[-:self:]" if you just wanted the dependencies, and "pip install myproj[-:self:,-:meta:,-:run:]" would become a really complicated way of saying "don't install anything at all". Contrast that with the current PEP 426 draft where the more common "just the dependencies please" operation is spelled "pip install myproj[-,:meta:,:run:]", while the "don't install anything at all" version gets the much shorter and more cryptic "pip install myproj[-]". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Sun Oct 12 14:36:19 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 12 Oct 2014 13:36:19 +0100 Subject: [Distutils] Having a "less complete" configuration for a project In-Reply-To: References: Message-ID: On 12 October 2014 13:04, Nick Coghlan wrote: >>> Any thoughts on how I could do this? >> >> I don't know of any current way to do it, and even the more flexible >> extras notation in PEP 426 doesn't quite get you there. >> >> The closest it currently allows is "pip install myproj[-,core]". The >> "-" disables installing the default dependencies, but it *also* >> disables installing the distribution itself. > > On second thoughts, I think PEP 426 does already cover this, by > letting you write "pip install myproj[-:meta:,-:run:,core]". That > would turn off the normal "meta_requires" and "run_requires" > processing, and just install the dependencies for the "core" extra > instead (which would then presumably be a subset of the normal > run_requires). > > You'd need to specify your core dependencies in two places (once in > run_requires and once for the core extra), but having setup.py as an > executable format copes with that OK. Thanks, that looks good enough for my needs. I must admit to not having really investigated this part of PEP 426, so thanks for the pointer. I'll go and do some reading. All I need to do now is wait for PEP 426 to get accepted, and a version of pip with support for it to be released :-) But I don't really mind ignoring the issue for now, until that happens. It's not like the project is even fully written yet... Paul From stefankrah at freenet.de Sun Oct 12 16:29:08 2014 From: stefankrah at freenet.de (Stefan Krah) Date: Sun, 12 Oct 2014 14:29:08 +0000 (UTC) Subject: [Distutils] some questions about PEP470 References: <20141009164139.GE7954@merlinux.eu> <89513014-4E0E-422D-AD12-95730EED60A8@stufft.io> <20141011063148.GN7954@merlinux.eu> <264E2355-E619-476E-992B-8D11BBBFB217@stufft.io> <12C9C976-9B22-4D29-9CD9-C9CC66576D9D@stufft.io> Message-ID: > (for example right now bytereef.org is down, so > we?d not discover any files there). Indeed. It was up reliably since 2005, down for maintenance on September 23rd (before ShellShock ...). Then I discovered that someone had put up m3-cdecimal on PyPI (presumably abusing PyPI as their private repo --- there are several m3-* packages now). This triggered some reflection on whether I would make a significant effort in the future to keep things running smoothly for an open source community where authors are largely viewed as expendable. Subsequently the downtime (again, the first one since 2005) was picked up for propagandistic purposes on Twitter and Reddit. Last year I would have felt an obligation to minimize the downtime to an hour at most. I no longer feel any such obligations and I'll do it when I have time. Stefan Krah From donald at stufft.io Sun Oct 12 20:27:15 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 12 Oct 2014 14:27:15 -0400 Subject: [Distutils] some questions about PEP470 In-Reply-To: References: <20141009164139.GE7954@merlinux.eu> <89513014-4E0E-422D-AD12-95730EED60A8@stufft.io> <20141011063148.GN7954@merlinux.eu> <264E2355-E619-476E-992B-8D11BBBFB217@stufft.io> <12C9C976-9B22-4D29-9CD9-C9CC66576D9D@stufft.io> Message-ID: <818DF4B5-FFAC-4F12-B0A0-3FF8970DF12D@stufft.io> > On Oct 12, 2014, at 10:29 AM, Stefan Krah wrote: > > >> (for example right now bytereef.org is down, so >> we?d not discover any files there). > > Indeed. It was up reliably since 2005, down for maintenance on > September 23rd (before ShellShock ...). Then I discovered that > someone had put up m3-cdecimal on PyPI (presumably abusing PyPI > as their private repo --- there are several m3-* packages now). If you?re this upset over someone redistributing your work, then maybe Open Source Software is the wrong hobby for you. Nonetheless I told you how to get that remediated and as of yet I don?t see an open support request from you on it. > > This triggered some reflection on whether I would make a significant > effort in the future to keep things running smoothly for an open source > community where authors are largely viewed as expendable. Authors are not viewed as expendable. Placing constraints on a system is just that, placing constraints on a system. > > > Subsequently the downtime (again, the first one since 2005) was picked > up for propagandistic purposes on Twitter and Reddit. I think you might want to rethink this strategy if it?s your goal, unless the view point you?re trying to push is that I was right all along because there are a number of people* who previously didn?t think it was a big deal but now agree with me since they couldn?t install cdecimal because bytrereef.org was down. * That have personally reached out to me. > > > Last year I would have felt an obligation to minimize the downtime > to an hour at most. I no longer feel any such obligations and I'll > do it when I have time. By all means! You don?t owe anyone your time, so if you want to wait until some other point go for it. Until then, well it?s making my case for me when people don?t understand why I care, so thanks :) --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From alex.gaynor at gmail.com Sun Oct 12 20:44:02 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Sun, 12 Oct 2014 18:44:02 +0000 (UTC) Subject: [Distutils] some questions about PEP470 References: <20141009164139.GE7954@merlinux.eu> <89513014-4E0E-422D-AD12-95730EED60A8@stufft.io> <20141011063148.GN7954@merlinux.eu> <264E2355-E619-476E-992B-8D11BBBFB217@stufft.io> <12C9C976-9B22-4D29-9CD9-C9CC66576D9D@stufft.io> Message-ID: Stefan Krah freenet.de> writes: > > > > (for example right now bytereef.org is down, so > > we?d not discover any files there). > > Indeed. It was up reliably since 2005, down for maintenance on > September 23rd (before ShellShock ...). Then I discovered that > someone had put up m3-cdecimal on PyPI (presumably abusing PyPI > as their private repo --- there are several m3-* packages now). > > This triggered some reflection on whether I would make a significant > effort in the future to keep things running smoothly for an open source > community where authors are largely viewed as expendable. I don't know what it means for "authors to be largely viewed as expendable", but half the point of hosting things on PyPI is that you *don't* need to do any work at all as an author for reliable delivery of your package. > > Subsequently the downtime (again, the first one since 2005) was picked > up for propagandistic purposes on Twitter and Reddit. Ok, but you seem to be doing the other side's propaganda. Every single person I've spoken to agrees that this just underscores the need to encourage packages to be on PyPI. > > Last year I would have felt an obligation to minimize the downtime > to an hour at most. I no longer feel any such obligations and I'll > do it when I have time. > Ok. The PyPI administrators still feel an obligation to their users, so I'll prefer packages under their care. > Stefan Krah > Cheers, Alex From graffatcolmingov at gmail.com Sun Oct 12 21:26:43 2014 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Sun, 12 Oct 2014 14:26:43 -0500 Subject: [Distutils] cdecimal licensing/hosting (was: some questions about PEP470) Message-ID: On Sun, Oct 12, 2014 at 1:44 PM, Alex Gaynor wrote: > Stefan Krah freenet.de> writes: > >> >> >> > (for example right now bytereef.org is down, so >> > we?d not discover any files there). >> >> Indeed. It was up reliably since 2005, down for maintenance on >> September 23rd (before ShellShock ...). Then I discovered that >> someone had put up m3-cdecimal on PyPI (presumably abusing PyPI >> as their private repo --- there are several m3-* packages now). >> >> This triggered some reflection on whether I would make a significant >> effort in the future to keep things running smoothly for an open source >> community where authors are largely viewed as expendable. > > I don't know what it means for "authors to be largely viewed as expendable", > but half the point of hosting things on PyPI is that you *don't* need to do any > work at all as an author for reliable delivery of your package. > >> >> Subsequently the downtime (again, the first one since 2005) was picked >> up for propagandistic purposes on Twitter and Reddit. > > Ok, but you seem to be doing the other side's propaganda. Every single person > I've spoken to agrees that this just underscores the need to encourage packages > to be on PyPI. > >> >> Last year I would have felt an obligation to minimize the downtime >> to an hour at most. I no longer feel any such obligations and I'll >> do it when I have time. >> > > Ok. The PyPI administrators still feel an obligation to their users, so I'll > prefer packages under their care. > >> Stefan Krah >> > > Cheers, > Alex Perhaps Stefan's referring to my tweets about the inability to reach bytereef but those weren't propaganda tweets. Those were tweets born out of utter frustration. Further, I'm rather shocked that you've decided to allow the site to remain unreachable because someone did what your license allowed them to do (redistribute the software while retaining the required information: copyright, license, etc). If you think that makes you expendable, you're half right. Users can redistribute your software, that's the nature of the license you chose to use. You're wrong because you, the author, are still very valuable to those very users who may encounter a bug in the future. I don't see how intentionally keeping your site unreachable does anything but hurt your users (unless of course you want them to redistribute it themselves or switch to Python 3.4). Does this mean that companies using devpi to keep an internal index that also have copies of cdecimal are somehow violating your rights? They're doing exactly what your license allows them to do. Or is it just that some group has decided to redistribute it directly through PyPI? I'm thoroughly confused here. From ncoghlan at gmail.com Sun Oct 12 23:58:39 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 13 Oct 2014 07:58:39 +1000 Subject: [Distutils] Having a "less complete" configuration for a project In-Reply-To: References: Message-ID: On 12 Oct 2014 22:36, "Paul Moore" wrote: > > On 12 October 2014 13:04, Nick Coghlan wrote: > >>> Any thoughts on how I could do this? > >> > >> I don't know of any current way to do it, and even the more flexible > >> extras notation in PEP 426 doesn't quite get you there. > >> > >> The closest it currently allows is "pip install myproj[-,core]". The > >> "-" disables installing the default dependencies, but it *also* > >> disables installing the distribution itself. > > > > On second thoughts, I think PEP 426 does already cover this, by > > letting you write "pip install myproj[-:meta:,-:run:,core]". That > > would turn off the normal "meta_requires" and "run_requires" > > processing, and just install the dependencies for the "core" extra > > instead (which would then presumably be a subset of the normal > > run_requires). > > > > You'd need to specify your core dependencies in two places (once in > > run_requires and once for the core extra), but having setup.py as an > > executable format copes with that OK. > > Thanks, that looks good enough for my needs. I must admit to not > having really investigated this part of PEP 426, so thanks for the > pointer. I'll go and do some reading. I don't think anyone other than me has really thought about it too much at this point - it's one of the areas in need of review and a draft reference implementation before the PEP could realistically be accepted. > All I need to do now is wait for PEP 426 to get accepted, and a > version of pip with support for it to be released :-) But I don't > really mind ignoring the issue for now, until that happens. It's not > like the project is even fully written yet... I think there's plenty of good upgrades on the agenda for pip 6 already, so I'll be nudging folks to start looking more closely at PEP 426 on the road to pip 7 :) Cheers, Nick. > > Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Mon Oct 13 12:40:02 2014 From: holger at merlinux.eu (holger krekel) Date: Mon, 13 Oct 2014 10:40:02 +0000 Subject: [Distutils] some questions about PEP470 In-Reply-To: <5439CFD2.5000909@oddbird.net> References: <20141009164139.GE7954@merlinux.eu> <89513014-4E0E-422D-AD12-95730EED60A8@stufft.io> <20141011063148.GN7954@merlinux.eu> <5439CFD2.5000909@oddbird.net> Message-ID: <20141013104002.GT7954@merlinux.eu> Hi Carl, Paul, all, On Sat, Oct 11, 2014 at 18:48 -0600, Carl Meyer wrote: > Hi Holger, > > On 10/11/2014 12:31 AM, holger krekel wrote: > > I understand that as a fairly generic security statement. But I was trying to > > rather ask about use cases and scenarios where precisely the > > --extra-index-url option is useful and to be recommended. > > I'd be grateful if Nick or you could still describe use cases, > > especially outside PEP470 external links context (the option existed > > before so i presume there must be some use cases). > > I don't use it anymore (because these days for everything other than > interactive playing around, I install only from a curated local index > specifically limited to each project's dependencies using --no-index and > --find-links), but I used to use it. My use case was this: generally > dependencies were installed from PyPI, but occasionally I would need to > patch a dependency, so I would create an sdist with a patched version > number (e.g. if I patched 2.0.1, I would create an sdist for version > 2.0.1.obc1, where "obc" is a tag based on my company name or the > project) and add this patched sdist to my own index, which I would add > to my installs with --extra-index-url. > > Because I used a patched version number and pinned all dependencies > exactly, it didn't matter to me that both PyPI and my "extra index" were > considered for installation; in fact that was convenient, since it meant > I could very easily upgrade to a newer PyPI release. > > I never used it for "private" non-PyPI packages. Right, makes sense and is in line with what Paul noted as his use case (adding wheels to existing pypi sdists): I think it's good and safe to use it when you are adding/patching things wrt existing projects on pypi.python.org. However, many people don't realize that using --extra-index-url to install private packages is a bad idea unless you register every private package as an empty pypi package. But the latter restriction is virtually never mentioned (and is an unrealistic recommendation in my opinion), see examples here: http://devcenter.gemfury.com/articles/pypi-server.html http://exhuma.github.io/mypi/index-config.html and I just noted that the very Python guide on packaging is advertising using plain --extra-index-url for private packages as well: http://docs.python-guide.org/en/latest/shipping/packaging/#personal-pypi and, besides the need for fixing the various discussions/pages, i think that PEP470 should contribute to a more careful discussion of the feature (it's fine for the actual external linking to existing pypi projects usecase, mind you). And i guess pip should have a warning note in the option help to help educating users. best, holger From p.f.moore at gmail.com Mon Oct 13 13:00:51 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 13 Oct 2014 12:00:51 +0100 Subject: [Distutils] some questions about PEP470 In-Reply-To: <20141013104002.GT7954@merlinux.eu> References: <20141009164139.GE7954@merlinux.eu> <89513014-4E0E-422D-AD12-95730EED60A8@stufft.io> <20141011063148.GN7954@merlinux.eu> <5439CFD2.5000909@oddbird.net> <20141013104002.GT7954@merlinux.eu> Message-ID: On 13 October 2014 11:40, holger krekel wrote: > and I just noted that the very Python guide on packaging is advertising > using plain --extra-index-url for private packages as well: > > http://docs.python-guide.org/en/latest/shipping/packaging/#personal-pypi I can see your point here (I'm not sure I agree with it, but that's a separate issue). If you want to propose a patch for the packaging user guide, we can discuss it there. > and, besides the need for fixing the various discussions/pages, i think > that PEP470 should contribute to a more careful discussion of the feature > (it's fine for the actual external linking to existing pypi projects > usecase, mind you). So if I read you correctly, you're saying that the PEP 470 usage of --extra-index-url is fine. That's good. I don't think it's the place of PEP 470 to discuss *other* uses of --extra-index-url. Having an example in there seemed fine to me, but if it brings up issues unrelated to the PEP then I think it's a distraction and should be removed. And I believe that's what has happened. So again, that's good. > And i guess pip should have a warning note in > the option help to help educating users. Again, not for the PEP, but feel free to raise a PR for pip (but once again, I reserve the right to disagree with that PR when it's raised :-)). Paul From holger at merlinux.eu Mon Oct 13 13:41:34 2014 From: holger at merlinux.eu (holger krekel) Date: Mon, 13 Oct 2014 11:41:34 +0000 Subject: [Distutils] some questions about PEP470 In-Reply-To: References: <20141009164139.GE7954@merlinux.eu> <89513014-4E0E-422D-AD12-95730EED60A8@stufft.io> <20141011063148.GN7954@merlinux.eu> <264E2355-E619-476E-992B-8D11BBBFB217@stufft.io> <12C9C976-9B22-4D29-9CD9-C9CC66576D9D@stufft.io> <8F2706F7-4804-4E20-8869-48FE075C2155@stufft.io> Message-ID: <20141013114134.GU7954@merlinux.eu> On Sun, Oct 12, 2014 at 10:10 +1000, Nick Coghlan wrote: > On 12 October 2014 09:49, Donald Stufft wrote: > > > > On Oct 11, 2014, at 7:48 PM, Nick Coghlan wrote: > > > >> On 12 October 2014 04:29, Donald Stufft wrote: > >>>> I plan to put the external repositories (and the commands needed to use them) > >>>> in the UI for PyPI. I suppose I should put that in the PEP as well, I was more > >>>> focused on defining the API differences and the changes. > >>> > >>> I forgot to mention, one other thing I thought about, we could move the current > >>> external links to external repositories automatically. I?m not sure if this is > >>> a good idea because on some projects that will end up with a giant list of things > >>> that a user would be suggested to install. Perhaps it?d work to scan them for > >>> installable files and only move the ones where an installable file was discovered, > >>> though that could be error prone (for example right now bytereef.org is down, so > >>> we?d not discover any files there). > >> > >> If I recall Holger's suggestion correctly, it was just to move the > >> current "Download URL" links over, rather than the scraped links. > > > > Hmm, I thought his suggestion was to just leave the existing links alone so that the > > existing pip/easy_installs would continue to function as they do now? > > Sure, we could do that too. Regardless, I don't believe we should let > the conversion of the silent security failures into noisy exceptions > block the other usability enhancements - if decoupling the two > decisions will achieve that, then let's go down that path. > > I think I suggested at one point that the PEP could potentially commit > to making the backwards compatible changes on the server side, and > then for the *in*compatible changes, only commit to *looking at the > numbers again* in a few months time after folks have had a chance to > experience the replacement model. Client applications would be > officially granted permission to ignore the PEP 438 external hosting > mechanism in favour of the new PEP 470 external index advertisements. > > The sequence of events from a compatibility break/user experience > enhancement perspective would then be: > > * new model added to PyPI (no compatibility break) > * old model dropped from pip (client side compatibility break, error > handling improvements) > * old model dropped from PyPI (conditional, based on changes in server metrics) I like this strategy and would add that I guess it's then only a matter of when not if the old model is dropped (and the PEP could encode that). The question is how to best do the "(almost) no compat break" for a reasonable definition of "almost". I am mostly concerned about the verified externals and "crawl urls". If they keep showing up as "rel=download" links on the simple page that should be fine for existing tools, right? But with PEP470 they might get an additional attribute so that a newer pip can recognize it and advise accordingly. How pypi/warehouse internally keeps the state during/post PEP470 is less clear to me but it might involve a one-time conversion from the existing information to the new one. I am actually wondering if a new pip could change its UI even without any changes to pypi.python.org? For example, when "pip install pil" happens it would say: Could not find any release file for "Pil" on https://pypi.python.org itself but the project has historically had additional external pages which linked to release files: http://effbot.org/zon/pil-changes-115.html/ http://www.pythonware.com/downloads/Imaging-1.1.3.tar.gz http://www.pythonware.com/products/pil Please manully check these links if they directly link to or contain release files for "Pil" and then add an appropriate one like this:: pip install --find-links URL [other options] Pil (See for more info ...) and pip could deduce this information from the current https://pypi.python.org/simple/pil/ page. (The pip message above may not be precisely right but you get the idea). FWIW this strategy of advising to use newer/other options but keeping the old working (undocumneted or documented with a warning) is how i generally try to handle things with pytest. It allows people to use the new way and incrementally migrate their setups. And if someone complains about problems with the old ways, point them to the new ways so there is no need to maintain the old ways too much. Even then, it would probably still makes sense to advance the server side and allow explicit registration of external index pages i guess (and maybe findlinks pages so you can really just use apache/nginx directly?) and then have pip provide a more precise message. > That changes the dynamic from "the updated user experience will be > better, trust us" to "if you prefer the old user experience, you can > keep using the old pip, if you prefer the new user experience, > upgrade". I think it's very much fine to say "this is the future" as long as old-tool/current-server instances keep working for a longer while. Python packaging infra is a bit like a big container ship these days -- which is both a good and a bad thing i guess -- and people build all kinds of funny containers on top. best, holger > The eventual decision on when to drop the legacy model from PyPI can > then by made based on a richer set of input data, that includes the > level of adoption of the new version of pip that treats all missing > link targets as potential errors. > > Regards, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > From holger at merlinux.eu Mon Oct 13 14:08:51 2014 From: holger at merlinux.eu (holger krekel) Date: Mon, 13 Oct 2014 12:08:51 +0000 Subject: [Distutils] some questions about PEP470 In-Reply-To: References: <20141009164139.GE7954@merlinux.eu> <89513014-4E0E-422D-AD12-95730EED60A8@stufft.io> <20141011063148.GN7954@merlinux.eu> <5439CFD2.5000909@oddbird.net> <20141013104002.GT7954@merlinux.eu> Message-ID: <20141013120851.GV7954@merlinux.eu> On Mon, Oct 13, 2014 at 12:00 +0100, Paul Moore wrote: > On 13 October 2014 11:40, holger krekel wrote: > > and I just noted that the very Python guide on packaging is advertising > > using plain --extra-index-url for private packages as well: > > > > http://docs.python-guide.org/en/latest/shipping/packaging/#personal-pypi > > I can see your point here (I'm not sure I agree with it, but that's a > separate issue). Sorry but what is there to agree or discuss? If recommending --extra-index-url for private packages does not come with a big fat warning that you need to publically register the name with PyPI, it exposes users to direct compromise of their machine, plain and simple. best, holger > If you want to propose a patch for the packaging user > guide, we can discuss it there. > > > and, besides the need for fixing the various discussions/pages, i think > > that PEP470 should contribute to a more careful discussion of the feature > > (it's fine for the actual external linking to existing pypi projects > > usecase, mind you). > > So if I read you correctly, you're saying that the PEP 470 usage of > --extra-index-url is fine. That's good. > > I don't think it's the place of PEP 470 to discuss *other* uses of > --extra-index-url. Having an example in there seemed fine to me, but > if it brings up issues unrelated to the PEP then I think it's a > distraction and should be removed. And I believe that's what has > happened. So again, that's good. > > > And i guess pip should have a warning note in > > the option help to help educating users. > > Again, not for the PEP, but feel free to raise a PR for pip (but once > again, I reserve the right to disagree with that PR when it's raised > :-)). > > Paul > From p.f.moore at gmail.com Mon Oct 13 14:37:49 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 13 Oct 2014 13:37:49 +0100 Subject: [Distutils] some questions about PEP470 In-Reply-To: <20141013120851.GV7954@merlinux.eu> References: <20141009164139.GE7954@merlinux.eu> <89513014-4E0E-422D-AD12-95730EED60A8@stufft.io> <20141011063148.GN7954@merlinux.eu> <5439CFD2.5000909@oddbird.net> <20141013104002.GT7954@merlinux.eu> <20141013120851.GV7954@merlinux.eu> Message-ID: On 13 October 2014 13:08, holger krekel wrote: > On Mon, Oct 13, 2014 at 12:00 +0100, Paul Moore wrote: >> On 13 October 2014 11:40, holger krekel wrote: >> > and I just noted that the very Python guide on packaging is advertising >> > using plain --extra-index-url for private packages as well: >> > >> > http://docs.python-guide.org/en/latest/shipping/packaging/#personal-pypi >> >> I can see your point here (I'm not sure I agree with it, but that's a >> separate issue). > > Sorry but what is there to agree or discuss? If recommending > --extra-index-url for private packages does not come with a big fat > warning that you need to publically register the name with PyPI, > it exposes users to direct compromise of their machine, plain and simple. I would view it as a matter of the trust model. If you don't trust PyPI, then you should not download direct from there. That applies whether or not you have a private index as well. if you do trust PyPI, then you presumably understand the risks. I'd be happy enough to see a note that whenever you use pip without specifying --no-index you trust PyPI. I don't mind if there's a further note that if you serve packages from your local index they will be considered as equal candidates with packages of the same name on PyPI *regardless of who uploaded them to PyPI*. But I don't accept that there's a need to over-stress the risk. After all, if I mistype an install command as "pip install devpy", I'm just as exposed to compromise of my machine. Paul. From donald at stufft.io Mon Oct 13 18:00:28 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 13 Oct 2014 12:00:28 -0400 Subject: [Distutils] some questions about PEP470 In-Reply-To: <20141013114134.GU7954@merlinux.eu> References: <20141009164139.GE7954@merlinux.eu> <89513014-4E0E-422D-AD12-95730EED60A8@stufft.io> <20141011063148.GN7954@merlinux.eu> <264E2355-E619-476E-992B-8D11BBBFB217@stufft.io> <12C9C976-9B22-4D29-9CD9-C9CC66576D9D@stufft.io> <8F2706F7-4804-4E20-8869-48FE075C2155@stufft.io> <20141013114134.GU7954@merlinux.eu> Message-ID: > On Oct 13, 2014, at 7:41 AM, holger krekel wrote: > > On Sun, Oct 12, 2014 at 10:10 +1000, Nick Coghlan wrote: >> On 12 October 2014 09:49, Donald Stufft wrote: >>> >>> On Oct 11, 2014, at 7:48 PM, Nick Coghlan wrote: >>> >>>> On 12 October 2014 04:29, Donald Stufft wrote: >>>>>> I plan to put the external repositories (and the commands needed to use them) >>>>>> in the UI for PyPI. I suppose I should put that in the PEP as well, I was more >>>>>> focused on defining the API differences and the changes. >>>>> >>>>> I forgot to mention, one other thing I thought about, we could move the current >>>>> external links to external repositories automatically. I?m not sure if this is >>>>> a good idea because on some projects that will end up with a giant list of things >>>>> that a user would be suggested to install. Perhaps it?d work to scan them for >>>>> installable files and only move the ones where an installable file was discovered, >>>>> though that could be error prone (for example right now bytereef.org is down, so >>>>> we?d not discover any files there). >>>> >>>> If I recall Holger's suggestion correctly, it was just to move the >>>> current "Download URL" links over, rather than the scraped links. >>> >>> Hmm, I thought his suggestion was to just leave the existing links alone so that the >>> existing pip/easy_installs would continue to function as they do now? >> >> Sure, we could do that too. Regardless, I don't believe we should let >> the conversion of the silent security failures into noisy exceptions >> block the other usability enhancements - if decoupling the two >> decisions will achieve that, then let's go down that path. >> >> I think I suggested at one point that the PEP could potentially commit >> to making the backwards compatible changes on the server side, and >> then for the *in*compatible changes, only commit to *looking at the >> numbers again* in a few months time after folks have had a chance to >> experience the replacement model. Client applications would be >> officially granted permission to ignore the PEP 438 external hosting >> mechanism in favour of the new PEP 470 external index advertisements. >> >> The sequence of events from a compatibility break/user experience >> enhancement perspective would then be: >> >> * new model added to PyPI (no compatibility break) >> * old model dropped from pip (client side compatibility break, error >> handling improvements) >> * old model dropped from PyPI (conditional, based on changes in server metrics) > > I like this strategy and would add that I guess it's then only a matter of > when not if the old model is dropped (and the PEP could encode that). This is more or less the sequence of events that will happen with PEP 470 as it is except that PEP 470 doesn?t make the deprecation optional. I do not believe it is a good idea to make it optional because it is a silent security risk. The other main difference is that PEP 470 doesn?t allow new projects to ever register externally hosted files in the old way. > > The question is how to best do the "(almost) no compat break" for > a reasonable definition of "almost". I am mostly concerned about > the verified externals and "crawl urls". If they keep showing up as > "rel=download" links on the simple page that should be fine for existing > tools, right? Keeping backwards compatibility for unsafe things is completely off the table as far as I?m concerned. Specifically that means the crawl links. I think keeping backwards compatibility for the ~70 projects which are doing things safely today is a waste of resources in the long run, but something that should be phased in as PEP 470 does. More people were broken when PyPI went forced TLS and then again when PyPI switched from a sha1 certificate to a sha256 certificate. The numbers of safe installs is miniscule. > > But with PEP470 they might get an additional attribute so that a newer > pip can recognize it and advise accordingly. How pypi/warehouse > internally keeps the state during/post PEP470 is less clear to me but it > might involve a one-time conversion from the existing information to the > new one. > > I am actually wondering if a new pip could change its UI even without > any changes to pypi.python.org? For example, when "pip install pil" > happens it would say: > > Could not find any release file for "Pil" on https://pypi.python.org > itself but the project has historically had additional external pages > which linked to release files: > > http://effbot.org/zon/pil-changes-115.html/ > http://www.pythonware.com/downloads/Imaging-1.1.3.tar.gz > http://www.pythonware.com/products/pil > > Please manully check these links if they directly link to or contain > release files for "Pil" and then add an appropriate one like this:: > > pip install --find-links URL [other options] Pil > > (See for more info ...) > > and pip could deduce this information from the current https://pypi.python.org/simple/pil/page. (The pip message above may not be precisely right but > you get the idea). > > FWIW this strategy of advising to use newer/other options but keeping > the old working (undocumneted or documented with a warning) > is how i generally try to handle things with pytest. It allows > people to use the new way and incrementally migrate their setups. > And if someone complains about problems with the old ways, point them to > the new ways so there is no need to maintain the old ways too much. It?s a fine way of doing thing, and we do it in pip too, when (and this is important), it?s not a security sensitive issue. > > Even then, it would probably still makes sense to advance the server > side and allow explicit registration of external index pages i guess > (and maybe findlinks pages so you can really just use apache/nginx > directly?) and then have pip provide a more precise message. > >> That changes the dynamic from "the updated user experience will be >> better, trust us" to "if you prefer the old user experience, you can >> keep using the old pip, if you prefer the new user experience, >> upgrade". > > I think it's very much fine to say "this is the future" as long as > old-tool/current-server instances keep working for a longer while. > > Python packaging infra is a bit like a big container ship these days -- > which is both a good and a bad thing i guess -- and people build > all kinds of funny containers on top. > > best, > holger > >> The eventual decision on when to drop the legacy model from PyPI can >> then by made based on a richer set of input data, that includes the >> level of adoption of the new version of pip that treats all missing >> link targets as potential errors. >> >> Regards, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From donald at stufft.io Mon Oct 13 20:23:33 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 13 Oct 2014 14:23:33 -0400 Subject: [Distutils] PEP 470 - Once More, with Feeling Message-ID: Alright, here's yet another go at PEP 470. See it online at www.python.org/dev/peps/pep-0470/ or reproduced in full down below. The diff between this version and the last is available at https://hg.python.org/peps/rev/2855fa903e89. Important Notes: * Continue to use a ```` link instead of an href to prevent older installers from silently picking up insecure hosting URLs. * Reduce the overall impact by dropping the special case for PIL and instead scan all projects for URLs which add installable files and move them to the new external repository feature. * Reduce the overall impact by explicitly stating that PyPI will add the location of any external repository in the UI for people using installers which have not implemented the discovery feature. * Explicitly call out the key user experience requirements of a solution to the general problem. * Simplify the ```` tag a bit, and also add explicit repository vs find-links types as well as include an example of the ``find-links`` type. * Allow a project to both host files on PyPI and register external repositories, these can be used for things which cannot be hosted on PyPI such as data files or Linux Wheels while still using PyPI as the repository for "regular" situations. * Mandate that the discovery mechanism must exist in a released pip prior to starting the deprecation process (with the exception of ``pypi-only`` for new projects) and call this out using an important admonition. * Explicitly call out the fact that 99.5% of the users of the deprecated features are doing so unsafely. * Explicitly reject the idea of preserving the existing links indefinentely. * Removed all examples which used the ``--extra-index-url`` feature of pip to remove the distraction of the discussion of how that currently works and in what scenarios it's safe or unsafe. Compatability: I've thought it over and gone back and forth on it to myself and others. I cannot justify an attempt to preserve backwards compatability when that backwards compatability is almost entirely unsafe to begin with. What I have done is remove the special case of PIL and essentially apply that to all projects. This should mean that all projects should have the correct metadata immediately without any need for interaction by the authors of said projects. I've also explicitly included adding the new metadata to the PyPI web UI to improve the discoverability for users of installers which don't have the discovery features. PEP: Abstract ======== This PEP proposes a mechanism for project authors to register with PyPI an external repository where their project's downloads can be located. This information can than be included as part of the simple API so that installers can use it to tell users where the item they are attempting to install is located and what they need to do to enable this additional repository. In addition to adding discovery information to make explicit multiple repositories easy to use, this PEP also deprecates and removes the implicit multiple repository support which currently functions through directly or indirectly linking off site via the simple API. Finally this PEP also proposes deprecating and removing the functionality added by PEP 438, particularly the additional rel information and the meta tag to indicate the API version. This PEP *does* not propose mandating that all authors upload their projects to PyPI in order to exist in the index nor does it propose any change to the human facing elements of PyPI. Rationale ========= Historically PyPI did not have any method of hosting files nor any method of automatically retrieving installables, it was instead focused on providing a central registry of names, to prevent naming collisions, and as a means of discovery for finding projects to use. In the course of time setuptools began to scrape these human facing pages, as well as pages linked from those pages, looking for things it could automatically download and install. Eventually this became the "Simple" API which used a similar URL structure however it eliminated any of the extraneous links and information to make the API more efficient. Additionally PyPI grew the ability for a project to upload release files directly to PyPI enabling PyPI to act as a repository in addition to an index. This gives PyPI two equally important roles that it plays in the Python ecosystem, that of index to enable easy discovery of Python projects and central repository to enable easy hosting, download, and installation of Python projects. Due to the history behind PyPI and the very organic growth it has experienced the lines between these two roles are blurry, and this blurring has caused confusion for the end users of both of these roles and this has in turn caused ire between people attempting to use PyPI in different capacities, most often when end users want to use PyPI as a repository but the author wants to use PyPI solely as an index. This confusion comes down to end users of projects not realizing if a project is hosted on PyPI or if it relies on an external service. This often manifests itself when the external service is down but PyPI is not. People will see that PyPI works, and other projects works, but this one specific one does not. They often times do not realize who they need to contact in order to get this fixed or what their remediation steps are. By moving to using explicit multiple repositories we can make the lines between these two roles much more explicit and remove the "hidden" surprises caused by the current implementation of handling people who do not want to use PyPI as a repository. However simply moving to explicit multiple repositories is a regression in discoverability, and for that reason this PEP adds an extension to the current simple API which will enable easy discovery of the specific repository that a project can be found in. PEP 438 attempted to solve this issue by allowing projects to explicitly declare if they were using the repository features or not, and if they were not, it had the installers classify the links it found as either "internal", "verifiable external" or "unverifiable external". PEP 438 was accepted and implemented in pip 1.4 (released on Jul 23, 2013) with the final transition implemented in pip 1.5 (released on Jan 2, 2014). PEP 438 was successful in bringing about more people to utilize PyPI's repository features, an altogether good thing given the global CDN powering PyPI providing speed ups for a lot of people, however it did so by introducing a new point of confusion and pain for both the end users and the authors. Key User Experience Expectations -------------------------------- #. Easily allow external hosting to "just work" when appropriately configured at the system, user or virtual environment level. #. Easily allow package authors to tell PyPI "my releases are hosted " and have that advertised in such a way that tools can clearly communicate it to users, without silently introducing unexpected dependencies on third party services. #. Eliminate any and all references to the confusing "verifiable external" and "unverifiable external" distinction from the user experience (both when installing and when releasing packages). #. The repository aspects of PyPI should become *just* the default package hosting location (i.e. the only one that is treated as opt-out rather than opt-in by most client tools in their default configuration). Aside from that aspect, hosting on PyPI should not otherwise provide an enhanced user experience over hosting your own package repository. #. Do all of the above while providing default behaviour that is secure against most attackers below the nation state adversary level. Why Additional Repositories? ---------------------------- The two common installer tools, pip and easy_install/setuptools, both support the concept of additional locations to search for files to satisfy the installation requirements and have done so for many years. This means that there is no need to "phase" in a new flag or concept and the solution to installing a project from a repository other than PyPI will function regardless of how old (within reason) the end user's installer is. Not only has this concept existed in the Python tooling for some time, but it is a concept that exists across languages and even extending to the OS level with OS package tools almost universally using multiple repository support making it extremely likely that someone is already familiar with the concept. Additionally, the multiple repository approach is a concept that is useful outside of the narrow scope of allowing projects which wish to be included on the index portion of PyPI but do not wish to utilize the repository portion of PyPI. This includes places where a company may wish to host a repository that contains their internal packages or where a project may wish to have multiple "channels" of releases, such as alpha, beta, release candidate, and final release. This could also be used for projects wishing to host files which cannot be uploaded to PyPI, such as multi-gigabyte data files or, currently at least, Linux Wheels. Why Not PEP 438 or Similar? --------------------------- While the additional search location support has existed in pip and setuptools for quite some time support for PEP 438 has only existed in pip since the 1.4 version, and still has yet to be implemented in setuptools. The design of PEP 438 did mean that users still benefited for projects which did not require external files even with older installers, however for projects which *did* require external files, users are still silently being given either potentially unreliable or, even worse, unsafe files to download. This system is also unique to Python as it arises out of the history of PyPI, this means that it is almost certain that this concept will be foreign to most, if not all users, until they encounter it while attempting to use the Python toolchain. Additionally, the classification system proposed by PEP 438 has, in practice, turned out to be extremely confusing to end users, so much so that it is a position of this PEP that the situation as it stands is completely untenable. The common pattern for a user with this system is to attempt to install a project possibly get an error message (or maybe not if the project ever uploaded something to PyPI but later switched without removing old files), see that the error message suggests ``--allow-external``, they reissue the command adding that flag most likely getting another error message, see that this time the error message suggests also adding ``--allow-unverified``, and again issue the command a third time, this time finally getting the thing they wish to install. This UX failure exists for several reasons. #. If pip can locate files at all for a project on the Simple API it will simply use that instead of attempting to locate more. This is generally the right thing to do as attempting to locate more would erase a large part of the benefit of PEP 438. This means that if a project *ever* uploaded a file that matches what the user has requested for install that will be used regardless of how old it is. #. PEP 438 makes an implicit assumption that most projects would either upload themselves to PyPI or would update themselves to directly linking to release files. While a large number of projects did ultimately decide to upload to PyPI, some of them did so only because the UX around what PEP 438 was so bad that they felt forced to do so. More concerning however, is the fact that very few projects have opted to directly and safely link to files and instead they still simply link to pages which must be scraped in order to find the actual files, thus rendering the safe variant (``--allow-external``) largely useless. #. Even if an author wishes to directly link to their files, doing so safely is non-obvious. It requires the inclusion of a MD5 hash (for historical reasons) in the hash of the URL. If they do not include this then their files will be considered "unverified". #. PEP 438 takes a security centric view and disallows any form of a global opt in for unverified projects. While this is generally a good thing, it creates extremely verbose and repetitive command invocations such as:: $ pip install --allow-external myproject --allow-unverified myproject myproject $ pip install --allow-all-external --allow-unverified myproject myproject Multiple Repository/Index Support ================================= Installers SHOULD implement or continue to offer, the ability to point the installer at multiple URL locations. The exact mechanisms for a user to indicate they wish to use an additional location is left up to each individual implementation. Additionally the mechanism discovering an installation candidate when multiple repositories are being used is also up to each individual implementation, however once configured an implementation should not discourage, warn, or otherwise cast a negative light upon the use of a repository simply because it is not the default repository. Currently both pip and setuptools implement multiple repository support by using the best installation candidate it can find from either repository, essentially treating it as if it were one large repository. Installers SHOULD also implement some mechanism for removing or otherwise disabling use of the default repository. The exact specifics of how that is achieved is up to each individual implementation. Installers SHOULD also implement some mechanism for whitelisting and blacklisting which projects a user wishes to install from a particular repository. The exact specifics of how that is achieved is up to each individual implementation. External Index Discovery ======================== One of the problems with using an additional index is one of discovery. Users will not generally be aware that an additional index is required at all much less where that index can be found. Projects can attempt to convey this information using their description on the PyPI page however that excludes people who discover their project organically through ``pip search``. To support projects that wish to externally host their files and to enable users to easily discover what additional indexes are required, PyPI will gain the ability for projects to register external index URLs along with an associated comment for each. These URLs will be made available on the simple page however they will not be linked or provided in a form that older installers will automatically search them. This ability will take the form of a ```` tag. The name of this tag must be set to ``repository`` or ``find-link`` and the content will be a link to the location of the repository. An optional data-description attribute will convey any comments or description that the author has provided. An example would look something like:: When an installer fetches the simple page for a project, if it finds this additional meta-data then it should use this data to tell the user how to add one or more of the additional URLs to search in. This message should include any comments that the project has included to enable them to communicate to the user and provide hints as to which URL they might want (e.g. if some are only useful or compatible with certain platforms or situations). When the installer has implemented the auto discovery mechanisms they should also deprecate any of the mechanisms added for PEP 438 (such as ``--allow-external``) for removal at the end of the deprecation period proposed by the PEP. In addition to the API for programtic access to the registered external repositories, PyPI will also prevent these URLs in the UI so that users with an installer that does not implement the discovery mechanism can still easily discover what repository the project is using to host itself. This feature **MUST** be added to PyPI and be contained in a released version of pip prior to starting the deprecation and removal process for the implicit offsite hosting functionality. Deprecation and Removal of Link Spidering ========================================= .. important:: The deprecation specified in this section **MUST** not start to until after the discovery mechanisms have been implemented and released in pip. The only exception to this is the addition of the ``pypi-only`` mode and defaulting new projects to it without abilility to switch to a different mode. A new hosting mode will be added to PyPI. This hosting mode will be called ``pypi-only`` and will be in addition to the three that PEP 438 has already given us which are ``pypi-explicit``, ``pypi-scrape``, ``pypi-scrape-crawl``. This new hosting mode will modify a project's simple api page so that it only lists the files which are directly hosted on PyPI and will not link to anything else. Upon acceptance of this PEP and the addition of the ``pypi-only`` mode, all new projects will be defaulted to the PyPI only mode and they will be locked to this mode and unable to change this particular setting. ``pypi-only`` projects will still be able to register external index URLs as described above - the "pypi-only" refers only to the download links that are published directly on PyPI. An email will then be sent out to all of the projects which are hosted only on PyPI informing them that in one month their project will be automatically converted to the ``pypi-only`` mode. A month after these emails have been sent any of those projects which were emailed, which still are hosted only on PyPI will have their mode set to ``pypi-only``. After that switch, an email will be sent to projects which rely on hosting external to PyPI. This email will warn these projects that externally hosted files have been deprecated on PyPI and that in 6 months from the time of that email that all external links will be removed from the installer APIs. This email **MUST** include instructions for converting their projects to be hosted on PyPI and **MUST** include links to a script or package that will enable them to enter their PyPI credentials and package name and have it automatically download and re-host all of their files on PyPI. This email **MUST** also include instructions for setting up their own index page and registering that with PyPI, including the fact that they can use pythonhosted.org as a host for an index page without requiring them to host any additional infrastructure or purchase a TLS certificate. This email must also contain a link to the Terms of Service for PyPI as many users may have signed up a long time ago and may not recall what those terms are. Finally this email must also contain a list of the links registered with PyPI where we were able to detect an installable file was located. Five months after the initial email, another email must be sent to any projects still relying on external hosting. This email will include all of the same information that the first email contained, except that the removal date will be one month away instead of six. Finally a month later all projects will be switched to the ``pypi-only`` mode and PyPI will be modified to remove the externally linked files functionality, when switching these projects to the ``pypi-only`` mode we will move any links which are able to be used for discovering other projects automatically to as an external repository. Summary of Changes ================== Repository side --------------- #. Implement simple API changes to allow the addition of an external repository. #. *(Optional, Mandatory on PyPI)* Deprecate and remove the hosting modes as defined by PEP 438. #. *(Optional, Mandatory on PyPI)* Restrict simple API to only list the files that are contained within the repository and the external repository metadata. Client side ----------- #. Implement multiple repository support. #. Implement some mechanism for removing/disabling the default repository. #. Implement the discovery mechanism. #. *(Optional)* Deprecate / Remove PEP 438 Impact ====== The large impact of this PEP will be that for users of older installation clients they will not get a discovery mechanism built into the install command. This will require them to browse to the PyPI web UI and discover the repository there. Since any URLs required to instal a project will be automatically migrated to the new format, the biggest change to users will be requiring a new option to install these projects. Looking at the numbers the actual impact should be quite low, with it affecting just 3.8% of projects which host any files only externally or 2.2% which have their latest version hosted only externally. 6674 unique IP addresses have accessed the Simple API for these 3.8% of projects in a single day (2014-09-30). Of those, 99.5% of them installed something which could not be verified, and thus they were open to a Remote Code Execution via a Man-In-The-Middle attack, while 7.9% installed something which could be verified and only 0.4% only installed things which could be verified. This means that 99.5% users of these features, both new and old, are doing something unsafe, and for anything using an older copy of pip or using setuptools at all they are silently unsafe. Projects Which Rely on Externally Hosted files ---------------------------------------------- This is determined by crawling the simple index and looking for installable files using a similar detection method as pip and setuptools use. The "latest" version is determined using ``pkg_resources.parse_version`` sort order and it is used to show whether or not the latest version is hosted externally or only old versions are. ============ ======= ================ =================== ======= \ PyPI External (old) External (latest) Total ============ ======= ================ =================== ======= **Safe** 43313 16 39 43368 **Unsafe** 0 756 1092 1848 **Total** 43313 772 1131 45216 ============ ======= ================ =================== ======= Top Externally Hosted Projects by Requests ------------------------------------------ This is determined by looking at the number of requests the ``/simple//`` page had gotten in a single day. The total number of requests during that day was 10,623,831. ============================== ======== Project Requests ============================== ======== PIL 63869 Pygame 2681 mysql-connector-python 1562 pyodbc 724 elementtree 635 salesforce-python-toolkit 316 wxPython 295 PyXML 251 RBTools 235 python-graph-core 123 cElementTree 121 ============================== ======== Top Externally Hosted Projects by Unique IPs -------------------------------------------- This is determined by looking at the IP addresses of requests the ``/simple//`` page had gotten in a single day. The total number of unique IP addresses during that day was 124,604. ============================== ========== Project Unique IPs ============================== ========== PIL 4553 mysql-connector-python 462 Pygame 202 pyodbc 181 elementtree 166 wxPython 126 RBTools 114 PyXML 87 salesforce-python-toolkit 76 pyDes 76 ============================== ========== Rejected Proposals ================== Keep the current classification system but adjust the options ------------------------------------------------------------- This PEP rejects several related proposals which attempt to fix some of the usability problems with the current system but while still keeping the general gist of PEP 438. This includes: * Default to allowing safely externally hosted files, but disallow unsafely hosted. * Default to disallowing safely externally hosted files with only a global flag to enable them, but disallow unsafely hosted. * Continue on the suggested path of PEP 438 and remove the option to unsafely host externally but continue to allow the option to safely host externally. These proposals are rejected because: * The classification system introduced in PEP 438 in an entirely unique concept to PyPI which is not generically applicable even in the context of Python packaging. Adding additional concepts comes at a cost. * The classification system itself is non-obvious to explain and to pre-determine what classification of link a project will require entails inspecting the project's ``/simple//`` page, and possibly any URLs linked from that page. * The ability to host externally while still being linked for automatic discovery is mostly a historic relic which causes a fair amount of pain and complexity for little reward. * The installer's ability to optimize or clean up the user interface is limited due to the nature of the implicit link scraping which would need to be done. This extends to the ``--allow-*`` options as well as the inability to determine if a link is expected to fail or not. * The mechanism paints a very broad brush when enabling an option, while PEP 438 attempts to limit this with per package options. However a project that has existed for an extended period of time may often times have several different URLs listed in their simple index. It is not unusual for at least one of these to no longer be under control of the project. While an unregistered domain will sit there relatively harmless most of the time, pip will continue to attempt to install from it on every discovery phase. This means that an attacker simply needs to look at projects which rely on unsafe external URLs and register expired domains to attack users. Implement this PEP, but Do Not Remove the Existing Links -------------------------------------------------------- This is essentially the backwards compatible version of this PEP. It attempts to allow people using older clients, or clients which do not implement this PEP to continue on as if nothing had changed. This proposal is rejected because the vast bulk of those scenarios are unsafe uses of the deprecated features. It is the opinion of this PEP that silently allowing unsafe actions to take place on behalf of end users is simply not an acceptable solution. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From donald at stufft.io Mon Oct 13 22:00:08 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 13 Oct 2014 16:00:08 -0400 Subject: [Distutils] PEP 470 - Once More, with Feeling In-Reply-To: References: Message-ID: <22C4BAD5-ED52-40A3-B2AB-65CB783ADCBE@stufft.io> > On Oct 13, 2014, at 2:23 PM, Donald Stufft wrote: > > Alright, here's yet another go at PEP 470. > > See it online at www.python.org/dev/peps/pep-0470/ or reproduced in full down > below. The diff between this version and the last is available at > https://hg.python.org/peps/rev/2855fa903e89. > > Important Notes: > > * Continue to use a ```` link instead of an href to prevent older > installers from silently picking up insecure hosting URLs. > > * Reduce the overall impact by dropping the special case for PIL and instead > scan all projects for URLs which add installable files and move them to > the new external repository feature. > > * Reduce the overall impact by explicitly stating that PyPI will add the > location of any external repository in the UI for people using installers > which have not implemented the discovery feature. > > * Explicitly call out the key user experience requirements of a solution to > the general problem. > > * Simplify the ```` tag a bit, and also add explicit repository vs > find-links types as well as include an example of the ``find-links`` type. > > * Allow a project to both host files on PyPI and register external repositories, > these can be used for things which cannot be hosted on PyPI such as data > files or Linux Wheels while still using PyPI as the repository for "regular" > situations. > > * Mandate that the discovery mechanism must exist in a released pip prior to > starting the deprecation process (with the exception of ``pypi-only`` for new > projects) and call this out using an important admonition. > > * Explicitly call out the fact that 99.5% of the users of the deprecated > features are doing so unsafely. > > * Explicitly reject the idea of preserving the existing links indefinentely. > > * Removed all examples which used the ``--extra-index-url`` feature of pip to > remove the distraction of the discussion of how that currently works and in > what scenarios it's safe or unsafe. > > > Compatability: > > I've thought it over and gone back and forth on it to myself and others. I > cannot justify an attempt to preserve backwards compatability when that > backwards compatability is almost entirely unsafe to begin with. What I have > done is remove the special case of PIL and essentially apply that to all > projects. This should mean that all projects should have the correct metadata > immediately without any need for interaction by the authors of said projects. > I've also explicitly included adding the new metadata to the PyPI web UI to > improve the discoverability for users of installers which don't have the > discovery features. > > PEP: > > [lots and lots of words] I forgot to mention, I?ve also added that installers should implement a feature by which you can white or blacklist specific projects from being installed from a particular repository. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From tleeuwenburg at gmail.com Tue Oct 14 03:40:55 2014 From: tleeuwenburg at gmail.com (Tennessee Leeuwenburg) Date: Tue, 14 Oct 2014 12:40:55 +1100 Subject: [Distutils] Process for taking over abandoned packages Message-ID: Hi all, I have just finished reading through most of the prior discussion on this issue, but as I read it in one sitting, I can't promise to have covered every nuance. I would like to propose that the pendulum has swung too far for some use cases. The use cases covered in that discussion appeared to me to be (a) Process for not allowing takeover a not-really/not-fully abandoned project (b) Process for taking over a (really) abandoned but useful project (c) Establishing graceful handover request processes and matchmaking I would like to suggest another edge case: (d) Taking over an abandoned project which does not have any code, hasn't been touched for years and doesn't do anything. That is to say, squatting. The projects which fill some use provide value to people because regardless of what maintainer politics may be happening, they are still installable, useful things. Squatter packages don't do any good to anyone. Mostly, these will have been created with good intentions to "get round to it" or otherwise provide some value to the community. However for whatever reason, they get left behind. If the original creator is still on email, then this can be handled through normal process and the wishes of the original author can be identified. However, I am in the middle of a situation where that's not the case, and the current formalities don't allow any wriggle room (as written, anyway). One possibility would be to allow take-over in non-contactable situations where the utility of the namespace is being wasted (subject to the administrator's interpretation). Another would be to use the "Development Status" flag to put different controls in place. Regardless, I am currently in a situation where I would like to register a particular name, and make use of it for at least somewhat valuable code. Someone registered the same name ages ago (over a year) and there's no code in it, and the github repo for the project is similarly inactive. They can't be reached. I've just posted to the issue tracker in case that reaches them, but my hopes are not high. I think it's a shame if the policy is written such that there is no recourse for exceptions and no way to avoid "locking in" cruft and accumulation of bad packages with no clear future use. I think it's also a reasonable requirement that maintainers and authors be contactable (within some definition of time period etc) In this case, I can consider another name, of course. Unfortunately (for me), this whole policy discussion has only come up after I started the process of looking into a pypi name takeover. I've put energy into setting up the project with its current name, and have now put a fair bit of time into trying to meet everyone's needs with regards to the PyPI entry. Changing the pypi name means changing all kinds of other information systems in order to have naming consistency, it's not just a trivial matter for me. It's hard for me to see how anyone is benefiting from this situation -- the current nameholder isn't actually doing anything, and it's just putting the brakes on getting things done and wasting namespace. I'd appreciate it if this group could consider putting some wording around handling exceptional circumstances, and/or dealing with "squatter" type scenarios. I realise I have simplified a few things, but I did so in the interest of getting to what I thought the most important aspects were. Let me know if you'd like any more information. Regards, -Tennessee -- -------------------------------------------------- Tennessee Leeuwenburg http://myownhat.blogspot.com/ "Don't believe everything you think" -------------- next part -------------- An HTML attachment was scrubbed... URL: From richard at python.org Tue Oct 14 04:38:15 2014 From: richard at python.org (Richard Jones) Date: Tue, 14 Oct 2014 13:38:15 +1100 Subject: [Distutils] Process for taking over abandoned packages In-Reply-To: References: Message-ID: Thanks for raising squatting as a concern. I have added what I think is a reasonable method of handling squatting (or otherwise unused name registrations): https://docs.google.com/document/d/1elum7ENjQb0dLB4ATfYNtnXYVLUzsKacc0VWnHHJb2A/edit?usp=sharing Richard On 14 October 2014 12:40, Tennessee Leeuwenburg wrote: > Hi all, > > I have just finished reading through most of the prior discussion on this > issue, but as I read it in one sitting, I can't promise to have covered > every nuance. I would like to propose that the pendulum has swung too far > for some use cases. > > The use cases covered in that discussion appeared to me to be > (a) Process for not allowing takeover a not-really/not-fully abandoned > project > (b) Process for taking over a (really) abandoned but useful project > (c) Establishing graceful handover request processes and matchmaking > > I would like to suggest another edge case: > (d) Taking over an abandoned project which does not have any code, > hasn't been touched for years and doesn't do anything. That is to say, > squatting. > > The projects which fill some use provide value to people because > regardless of what maintainer politics may be happening, they are still > installable, useful things. > > Squatter packages don't do any good to anyone. Mostly, these will have > been created with good intentions to "get round to it" or otherwise provide > some value to the community. However for whatever reason, they get left > behind. If the original creator is still on email, then this can be handled > through normal process and the wishes of the original author can be > identified. However, I am in the middle of a situation where that's not the > case, and the current formalities don't allow any wriggle room (as written, > anyway). > > One possibility would be to allow take-over in non-contactable situations > where the utility of the namespace is being wasted (subject to the > administrator's interpretation). Another would be to use the "Development > Status" flag to put different controls in place. > > Regardless, I am currently in a situation where I would like to register a > particular name, and make use of it for at least somewhat valuable code. > Someone registered the same name ages ago (over a year) and there's no code > in it, and the github repo for the project is similarly inactive. They > can't be reached. I've just posted to the issue tracker in case that > reaches them, but my hopes are not high. > > I think it's a shame if the policy is written such that there is no > recourse for exceptions and no way to avoid "locking in" cruft and > accumulation of bad packages with no clear future use. I think it's also a > reasonable requirement that maintainers and authors be contactable (within > some definition of time period etc) > > In this case, I can consider another name, of course. Unfortunately (for > me), this whole policy discussion has only come up after I started the > process of looking into a pypi name takeover. I've put energy into setting > up the project with its current name, and have now put a fair bit of time > into trying to meet everyone's needs with regards to the PyPI entry. > Changing the pypi name means changing all kinds of other information > systems in order to have naming consistency, it's not just a trivial matter > for me. It's hard for me to see how anyone is benefiting from this > situation -- the current nameholder isn't actually doing anything, and it's > just putting the brakes on getting things done and wasting namespace. > > I'd appreciate it if this group could consider putting some wording around > handling exceptional circumstances, and/or dealing with "squatter" type > scenarios. > > I realise I have simplified a few things, but I did so in the interest of > getting to what I thought the most important aspects were. Let me know if > you'd like any more information. > > Regards, > -Tennessee > > > -- > -------------------------------------------------- > Tennessee Leeuwenburg > http://myownhat.blogspot.com/ > "Don't believe everything you think" > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.eu Tue Oct 14 09:54:30 2014 From: holger at merlinux.eu (holger krekel) Date: Tue, 14 Oct 2014 07:54:30 +0000 Subject: [Distutils] Process for taking over abandoned packages In-Reply-To: References: Message-ID: <20141014075430.GX7954@merlinux.eu> On Tue, Oct 14, 2014 at 13:38 +1100, Richard Jones wrote: > Thanks for raising squatting as a concern. I have added what I think is a > reasonable method of handling squatting (or otherwise unused name > registrations): > > https://docs.google.com/document/d/1elum7ENjQb0dLB4ATfYNtnXYVLUzsKacc0VWnHHJb2A/edit?usp=sharing thanks for going through the effort! I added some suggestions - do you see them in the doc? Also i'd suggest to change the order of sections, first "Proposed Process" and then Conditions of Transfer. It's a bit unclear to me how someone starts the process, i.e. applies for getting a name that is currently taken. What about requiring that an Issue needs to be opened with the pypi tracker and the admin who does the (forceful) transfer shall close it approprirately so that we also have some permanent trail of what happened/was tried/discussed? best, holger > > Richard > > On 14 October 2014 12:40, Tennessee Leeuwenburg > wrote: > > > Hi all, > > > > I have just finished reading through most of the prior discussion on this > > issue, but as I read it in one sitting, I can't promise to have covered > > every nuance. I would like to propose that the pendulum has swung too far > > for some use cases. > > > > The use cases covered in that discussion appeared to me to be > > (a) Process for not allowing takeover a not-really/not-fully abandoned > > project > > (b) Process for taking over a (really) abandoned but useful project > > (c) Establishing graceful handover request processes and matchmaking > > > > I would like to suggest another edge case: > > (d) Taking over an abandoned project which does not have any code, > > hasn't been touched for years and doesn't do anything. That is to say, > > squatting. > > > > The projects which fill some use provide value to people because > > regardless of what maintainer politics may be happening, they are still > > installable, useful things. > > > > Squatter packages don't do any good to anyone. Mostly, these will have > > been created with good intentions to "get round to it" or otherwise provide > > some value to the community. However for whatever reason, they get left > > behind. If the original creator is still on email, then this can be handled > > through normal process and the wishes of the original author can be > > identified. However, I am in the middle of a situation where that's not the > > case, and the current formalities don't allow any wriggle room (as written, > > anyway). > > > > One possibility would be to allow take-over in non-contactable situations > > where the utility of the namespace is being wasted (subject to the > > administrator's interpretation). Another would be to use the "Development > > Status" flag to put different controls in place. > > > > Regardless, I am currently in a situation where I would like to register a > > particular name, and make use of it for at least somewhat valuable code. > > Someone registered the same name ages ago (over a year) and there's no code > > in it, and the github repo for the project is similarly inactive. They > > can't be reached. I've just posted to the issue tracker in case that > > reaches them, but my hopes are not high. > > > > I think it's a shame if the policy is written such that there is no > > recourse for exceptions and no way to avoid "locking in" cruft and > > accumulation of bad packages with no clear future use. I think it's also a > > reasonable requirement that maintainers and authors be contactable (within > > some definition of time period etc) > > > > In this case, I can consider another name, of course. Unfortunately (for > > me), this whole policy discussion has only come up after I started the > > process of looking into a pypi name takeover. I've put energy into setting > > up the project with its current name, and have now put a fair bit of time > > into trying to meet everyone's needs with regards to the PyPI entry. > > Changing the pypi name means changing all kinds of other information > > systems in order to have naming consistency, it's not just a trivial matter > > for me. It's hard for me to see how anyone is benefiting from this > > situation -- the current nameholder isn't actually doing anything, and it's > > just putting the brakes on getting things done and wasting namespace. > > > > I'd appreciate it if this group could consider putting some wording around > > handling exceptional circumstances, and/or dealing with "squatter" type > > scenarios. > > > > I realise I have simplified a few things, but I did so in the interest of > > getting to what I thought the most important aspects were. Let me know if > > you'd like any more information. > > > > Regards, > > -Tennessee > > > > > > -- > > -------------------------------------------------- > > Tennessee Leeuwenburg > > http://myownhat.blogspot.com/ > > "Don't believe everything you think" > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From robin at reportlab.com Tue Oct 14 11:37:30 2014 From: robin at reportlab.com (Robin Becker) Date: Tue, 14 Oct 2014 10:37:30 +0100 Subject: [Distutils] setuptools script installation in Mac OS Message-ID: <543CEEDA.90905@chamonix.reportlab.co.uk> I have a setuptools based setup.py that contains this > entry_points = dict( > console_scripts = [ > 'rml2pdf=rlextra.rml2pdf.rml2pdf:main', > 'pageCatcher=rlextra.pageCatcher.pageCatcher:scriptInterp', > 'pdfexplorer=rlextra.pageCatcher.pdfexplorer:test', > ], > gui_scripts = [ > 'diagra=rlextra.graphics.guiedit.guiedit:mainApp', > ], > ), this appears to work OK and creates all 4 scripts in Windows, Linux & Mac OS, however, when uninstalling or upgrading I get an error for Mac OS when instead of trying to remove a pageCatcher script it tries instead to remove pagecatcher. ie > > Proceed (y/n)? y > Exception: > Traceback (most recent call last): > File "/Users/rptlab/tmp/ttt/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main > status = self.run(options, args) > File "/Users/rptlab/tmp/ttt/lib/python2.7/site-packages/pip/commands/uninstall.py", line 59, in run > requirement_set.uninstall(auto_confirm=options.yes) > File "/Users/rptlab/tmp/ttt/lib/python2.7/site-packages/pip/req.py", line 1024, in uninstall > req.uninstall(auto_confirm=auto_confirm) > File "/Users/rptlab/tmp/ttt/lib/python2.7/site-packages/pip/req.py", line 594, in uninstall > paths_to_remove.remove(auto_confirm) > File "/Users/rptlab/tmp/ttt/lib/python2.7/site-packages/pip/req.py", line 1781, in remove > renames(path, new_path) > File "/Users/rptlab/tmp/ttt/lib/python2.7/site-packages/pip/util.py", line 295, in renames > shutil.move(old, new) > File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 301, in move > copy2(src, real_dst) > File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 130, in copy2 > copyfile(src, dst) > File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 82, in copyfile > with open(src, 'rb') as fsrc: > IOError: [Errno 2] No such file or directory: '/Users/rptlab/tmp/ttt/bin/pagecatcher' > > Storing debug log for failure in /Users/rptlab/.pip/pip.log this is with pip 1.5.2 & python 2.7.6 in Darwin 10.8.0 and other later Macs. Is this just a case issue? -- Robin Becker From mal at egenix.com Tue Oct 14 21:44:58 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 14 Oct 2014 21:44:58 +0200 Subject: [Distutils] cdecimal licensing/hosting (was: some questions about PEP470) In-Reply-To: References: Message-ID: <543D7D3A.3010307@egenix.com> Gentlemen, could you please stop this and show some more respect in these discussions ? Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ On 12.10.2014 21:26, Ian Cordasco wrote: > On Sun, Oct 12, 2014 at 1:44 PM, Alex Gaynor wrote: >> Stefan Krah freenet.de> writes: >> >>> >>> >>>> (for example right now bytereef.org is down, so >>>> we?d not discover any files there). >>> >>> Indeed. It was up reliably since 2005, down for maintenance on >>> September 23rd (before ShellShock ...). Then I discovered that >>> someone had put up m3-cdecimal on PyPI (presumably abusing PyPI >>> as their private repo --- there are several m3-* packages now). >>> >>> This triggered some reflection on whether I would make a significant >>> effort in the future to keep things running smoothly for an open source >>> community where authors are largely viewed as expendable. >> >> I don't know what it means for "authors to be largely viewed as expendable", >> but half the point of hosting things on PyPI is that you *don't* need to do any >> work at all as an author for reliable delivery of your package. >> >>> >>> Subsequently the downtime (again, the first one since 2005) was picked >>> up for propagandistic purposes on Twitter and Reddit. >> >> Ok, but you seem to be doing the other side's propaganda. Every single person >> I've spoken to agrees that this just underscores the need to encourage packages >> to be on PyPI. >> >>> >>> Last year I would have felt an obligation to minimize the downtime >>> to an hour at most. I no longer feel any such obligations and I'll >>> do it when I have time. >>> >> >> Ok. The PyPI administrators still feel an obligation to their users, so I'll >> prefer packages under their care. >> >>> Stefan Krah >>> >> >> Cheers, >> Alex > > Perhaps Stefan's referring to my tweets about the inability to reach > bytereef but those weren't propaganda tweets. Those were tweets born > out of utter frustration. Further, I'm rather shocked that you've > decided to allow the site to remain unreachable because someone did > what your license allowed them to do (redistribute the software while > retaining the required information: copyright, license, etc). If you > think that makes you expendable, you're half right. Users can > redistribute your software, that's the nature of the license you chose > to use. You're wrong because you, the author, are still very valuable > to those very users who may encounter a bug in the future. I don't see > how intentionally keeping your site unreachable does anything but hurt > your users (unless of course you want them to redistribute it > themselves or switch to Python 3.4). > > Does this mean that companies using devpi to keep an internal index > that also have copies of cdecimal are somehow violating your rights? > They're doing exactly what your license allows them to do. Or is it > just that some group has decided to redistribute it directly through > PyPI? I'm thoroughly confused here. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > From richard at python.org Wed Oct 15 00:08:48 2014 From: richard at python.org (Richard Jones) Date: Wed, 15 Oct 2014 09:08:48 +1100 Subject: [Distutils] Process for taking over abandoned packages In-Reply-To: <20141014075430.GX7954@merlinux.eu> References: <20141014075430.GX7954@merlinux.eu> Message-ID: Yep, I get notified of comments, thanks. I'll add information on how to start the process - I always require an issue in the support tracker. On 14 October 2014 18:54, holger krekel wrote: > On Tue, Oct 14, 2014 at 13:38 +1100, Richard Jones wrote: > > Thanks for raising squatting as a concern. I have added what I think is a > > reasonable method of handling squatting (or otherwise unused name > > registrations): > > > > > https://docs.google.com/document/d/1elum7ENjQb0dLB4ATfYNtnXYVLUzsKacc0VWnHHJb2A/edit?usp=sharing > > thanks for going through the effort! > > I added some suggestions - do you see them in the doc? > > Also i'd suggest to change the order of sections, first "Proposed Process" > and then Conditions of Transfer. > > It's a bit unclear to me how someone starts the process, i.e. applies > for getting a name that is currently taken. What about requiring that > an Issue needs to be opened with the pypi tracker and the admin who does > the (forceful) transfer shall close it approprirately so that we also > have some permanent trail of what happened/was tried/discussed? > > best, > holger > > > > > > Richard > > > > On 14 October 2014 12:40, Tennessee Leeuwenburg > > wrote: > > > > > Hi all, > > > > > > I have just finished reading through most of the prior discussion on > this > > > issue, but as I read it in one sitting, I can't promise to have covered > > > every nuance. I would like to propose that the pendulum has swung too > far > > > for some use cases. > > > > > > The use cases covered in that discussion appeared to me to be > > > (a) Process for not allowing takeover a not-really/not-fully > abandoned > > > project > > > (b) Process for taking over a (really) abandoned but useful project > > > (c) Establishing graceful handover request processes and matchmaking > > > > > > I would like to suggest another edge case: > > > (d) Taking over an abandoned project which does not have any code, > > > hasn't been touched for years and doesn't do anything. That is to say, > > > squatting. > > > > > > The projects which fill some use provide value to people because > > > regardless of what maintainer politics may be happening, they are still > > > installable, useful things. > > > > > > Squatter packages don't do any good to anyone. Mostly, these will have > > > been created with good intentions to "get round to it" or otherwise > provide > > > some value to the community. However for whatever reason, they get left > > > behind. If the original creator is still on email, then this can be > handled > > > through normal process and the wishes of the original author can be > > > identified. However, I am in the middle of a situation where that's > not the > > > case, and the current formalities don't allow any wriggle room (as > written, > > > anyway). > > > > > > One possibility would be to allow take-over in non-contactable > situations > > > where the utility of the namespace is being wasted (subject to the > > > administrator's interpretation). Another would be to use the > "Development > > > Status" flag to put different controls in place. > > > > > > Regardless, I am currently in a situation where I would like to > register a > > > particular name, and make use of it for at least somewhat valuable > code. > > > Someone registered the same name ages ago (over a year) and there's no > code > > > in it, and the github repo for the project is similarly inactive. They > > > can't be reached. I've just posted to the issue tracker in case that > > > reaches them, but my hopes are not high. > > > > > > I think it's a shame if the policy is written such that there is no > > > recourse for exceptions and no way to avoid "locking in" cruft and > > > accumulation of bad packages with no clear future use. I think it's > also a > > > reasonable requirement that maintainers and authors be contactable > (within > > > some definition of time period etc) > > > > > > In this case, I can consider another name, of course. Unfortunately > (for > > > me), this whole policy discussion has only come up after I started the > > > process of looking into a pypi name takeover. I've put energy into > setting > > > up the project with its current name, and have now put a fair bit of > time > > > into trying to meet everyone's needs with regards to the PyPI entry. > > > Changing the pypi name means changing all kinds of other information > > > systems in order to have naming consistency, it's not just a trivial > matter > > > for me. It's hard for me to see how anyone is benefiting from this > > > situation -- the current nameholder isn't actually doing anything, and > it's > > > just putting the brakes on getting things done and wasting namespace. > > > > > > I'd appreciate it if this group could consider putting some wording > around > > > handling exceptional circumstances, and/or dealing with "squatter" type > > > scenarios. > > > > > > I realise I have simplified a few things, but I did so in the interest > of > > > getting to what I thought the most important aspects were. Let me know > if > > > you'd like any more information. > > > > > > Regards, > > > -Tennessee > > > > > > > > > -- > > > -------------------------------------------------- > > > Tennessee Leeuwenburg > > > http://myownhat.blogspot.com/ > > > "Don't believe everything you think" > > > > > > _______________________________________________ > > > Distutils-SIG maillist - Distutils-SIG at python.org > > > https://mail.python.org/mailman/listinfo/distutils-sig > > > > > > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefankrah at freenet.de Wed Oct 15 02:50:45 2014 From: stefankrah at freenet.de (Stefan Krah) Date: Wed, 15 Oct 2014 00:50:45 +0000 (UTC) Subject: [Distutils] some questions about PEP470 References: <20141009164139.GE7954@merlinux.eu> <89513014-4E0E-422D-AD12-95730EED60A8@stufft.io> <20141011063148.GN7954@merlinux.eu> <264E2355-E619-476E-992B-8D11BBBFB217@stufft.io> <12C9C976-9B22-4D29-9CD9-C9CC66576D9D@stufft.io> <818DF4B5-FFAC-4F12-B0A0-3FF8970DF12D@stufft.io> Message-ID: Donald Stufft stufft.io> writes: > If you're this upset over someone redistributing your work, then maybe > Open Source Software is the wrong hobby for you. Usually one does not tell a core developer that his contributions are "a hobby". I have contributed 40000+ lines of original, dense C code, backed by 100% code coverage and 30000+ lines of ACL2 proofs. These days it may be more productive to hack other people's brains and produce 10000+ tweets in order to have more political influence. It's great that my code is distributed with Python. Likewise, it was great to work with Matthias Klose and Hans-Peter Jansen to produce Debian, Ubuntu, and OpenSuse packages. cdecimal is also distributed by NetBSD, ActiveState, continuum.io, and others. The difference here is that the above persons and entities respect people. They respect software. The package maintainers are very competent and easy to work with. > Nonetheless I told you how to get that remediated and as of yet I don't > see an open support request from you on it. My interest in cleaning up PyPI is practically zero now. In the end, who cares what m3-cdecimal was supposed to accomplish: Maybe they wanted to teach me a lesson, maybe they were upset about a minor detail, maybe they have a zero-day exploit for tar and that's their delivery mode. All I know is that they didn't even run ``make distclean'' before packaging, so some user info is present in config.log. Some problems can only be fixed by a curated distribution run by neutral maintainers. > I think you might want to rethink this strategy if it's your goal, unless > the view point you're trying to push is that I was right all along because > there are a number of people* who previously didn't think it was a big deal > but now agree with me since they couldn't install cdecimal because bytrereef.org > was down. Shrug. This is more loss of interest than a strike! Even it were a strike, the observation is not particularly interesting: Any strike (think railway) potentially alienates some users. Anyway, it will be kind of tough to force U.S. exceptionalism via the terms and conditions on an international body of authors if only uploaded packages are allowed. Stefan Krah From donald at stufft.io Wed Oct 15 03:15:43 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 14 Oct 2014 21:15:43 -0400 Subject: [Distutils] some questions about PEP470 In-Reply-To: References: <20141009164139.GE7954@merlinux.eu> <89513014-4E0E-422D-AD12-95730EED60A8@stufft.io> <20141011063148.GN7954@merlinux.eu> <264E2355-E619-476E-992B-8D11BBBFB217@stufft.io> <12C9C976-9B22-4D29-9CD9-C9CC66576D9D@stufft.io> <818DF4B5-FFAC-4F12-B0A0-3FF8970DF12D@stufft.io> Message-ID: On Oct 14, 2014, at 8:50 PM, Stefan Krah wrote: > Donald Stufft stufft.io> writes: > >> If you're this upset over someone redistributing your work, then maybe >> Open Source Software is the wrong hobby for you. > > Usually one does not tell a core developer that his contributions are > "a hobby". I have contributed 40000+ lines of original, dense C code, > backed by 100% code coverage and 30000+ lines of ACL2 proofs. Uhh, maybe you?re misunderstanding the word hobby, unless you?re getting paid for your OSS work you?re not doing it professionally. A hobby isn?t a negative thing, until last December my OSS work was entirely a hobby too, and it?s still a hobby in my spare time too. > > These days it may be more productive to hack other people's brains and > produce 10000+ tweets in order to have more political influence. I?m not even sure what this is saying? Are you accusing me of hacking people?s brains? If so I?m kind of impressed by what power you think I have. > > > It's great that my code is distributed with Python. Likewise, it was great > to work with Matthias Klose and Hans-Peter Jansen to produce Debian, Ubuntu, > and OpenSuse packages. cdecimal is also distributed by NetBSD, ActiveState, > continuum.io, and others. > > > The difference here is that the above persons and entities respect people. > They respect software. The package maintainers are very competent and easy > to work with. If this is about m3-cdecimal, well your license doesn?t give it?s permissions out only if the person is nice to you. > > >> Nonetheless I told you how to get that remediated and as of yet I don't >> see an open support request from you on it. > > My interest in cleaning up PyPI is practically zero now. In the end, who > cares what m3-cdecimal was supposed to accomplish: Maybe they wanted to > teach me a lesson, maybe they were upset about a minor detail, maybe they > have a zero-day exploit for tar and that's their delivery mode. Ok, if you don?t care then I find it hard to care much about it either. > > All I know is that they didn't even run ``make distclean'' before packaging, > so some user info is present in config.log. > > > Some problems can only be fixed by a curated distribution run by neutral > maintainers. Sure. > > >> I think you might want to rethink this strategy if it's your goal, unless >> the view point you're trying to push is that I was right all along because >> there are a number of people* who previously didn't think it was a big deal >> but now agree with me since they couldn't install cdecimal because > bytrereef.org >> was down. > > > Shrug. This is more loss of interest than a strike! Even it were a strike, > the observation is not particularly interesting: Any strike (think railway) > potentially alienates some users. Ok! Well a loss of interest makes for a good example too, so thanks! > > Anyway, it will be kind of tough to force U.S. exceptionalism via the terms > and conditions on an international body of authors if only uploaded packages > are allowed. > I?m not even sure what this is trying to say? How are our pretty simple ToS some sort of US exceptionalism? > > > Stefan Krah > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From ncoghlan at gmail.com Wed Oct 15 03:33:50 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 15 Oct 2014 11:33:50 +1000 Subject: [Distutils] some questions about PEP470 In-Reply-To: References: <20141009164139.GE7954@merlinux.eu> <89513014-4E0E-422D-AD12-95730EED60A8@stufft.io> <20141011063148.GN7954@merlinux.eu> <264E2355-E619-476E-992B-8D11BBBFB217@stufft.io> <12C9C976-9B22-4D29-9CD9-C9CC66576D9D@stufft.io> <818DF4B5-FFAC-4F12-B0A0-3FF8970DF12D@stufft.io> Message-ID: On 15 Oct 2014 11:16, "Donald Stufft" wrote: > On Oct 14, 2014, at 8:50 PM, Stefan Krah wrote: > > > > > Anyway, it will be kind of tough to force U.S. exceptionalism via the terms > > and conditions on an international body of authors if only uploaded packages > > are allowed. > > > > I?m not even sure what this is trying to say? How are our pretty simple ToS > some sort of US exceptionalism? PyPI is hosted in the US, and thus covered by US export laws. I don't follow Stefan's objection, however, given that the objective of PEP 470 is to improve the user experience of external hosting, rather than to disallow it. We're also working with the TUF developers to make sure that the next draft of their PEP appropriately covers the external hosting use case. The only things we're actively trying to eliminate are the MITM vulnerability affecting the majority of current externally hosted packages, and the poor user experience that arises when the current link spidering mechanism leads to packaging clients feeling obliged to silently ignore unreachable URLs when looking for externally hosted packages. Regards, Nick. > > > > > > > Stefan Krah > > > > > > > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefankrah at freenet.de Wed Oct 15 03:59:40 2014 From: stefankrah at freenet.de (Stefan Krah) Date: Wed, 15 Oct 2014 01:59:40 +0000 (UTC) Subject: [Distutils] some questions about PEP470 References: <20141009164139.GE7954@merlinux.eu> <89513014-4E0E-422D-AD12-95730EED60A8@stufft.io> <20141011063148.GN7954@merlinux.eu> <264E2355-E619-476E-992B-8D11BBBFB217@stufft.io> <12C9C976-9B22-4D29-9CD9-C9CC66576D9D@stufft.io> <818DF4B5-FFAC-4F12-B0A0-3FF8970DF12D@stufft.io> Message-ID: Nick Coghlan gmail.com> writes: > PyPI is hosted in the US, and thus covered by US export laws. > I don't follow Stefan's objection, however, given that the objective of PEP 470 is to improve the user experience of external hosting, rather than to disallow it. Sorry if it wasn't clear. This sub-thread was no longer about PEP 470 at all. Rough context: Mr. Stufft: cdecimal downtime means more people will want to enforce uploading. Me: Even if that is so, you cannot exploit it politically, since upload-only packages are an insult for international authors. Stefan Krah From stefankrah at freenet.de Wed Oct 15 04:20:22 2014 From: stefankrah at freenet.de (Stefan Krah) Date: Wed, 15 Oct 2014 02:20:22 +0000 (UTC) Subject: [Distutils] some questions about PEP470 References: <20141009164139.GE7954@merlinux.eu> <89513014-4E0E-422D-AD12-95730EED60A8@stufft.io> <20141011063148.GN7954@merlinux.eu> <264E2355-E619-476E-992B-8D11BBBFB217@stufft.io> <12C9C976-9B22-4D29-9CD9-C9CC66576D9D@stufft.io> <818DF4B5-FFAC-4F12-B0A0-3FF8970DF12D@stufft.io> Message-ID: At this point (and possibly before) you are just trolling and not worth any further correspondence. If some of your feigned surprise questions are actually genuine, I recommend walking away from the keyboard for a couple of weeks and reading some literature. Otherwise it is just a waste of time to attempt any further conversation. From nad at acm.org Wed Oct 15 06:58:33 2014 From: nad at acm.org (Ned Deily) Date: Tue, 14 Oct 2014 21:58:33 -0700 Subject: [Distutils] setuptools script installation in Mac OS References: <543CEEDA.90905@chamonix.reportlab.co.uk> Message-ID: In article <543CEEDA.90905 at chamonix.reportlab.co.uk>, Robin Becker wrote: > I have a setuptools based setup.py that contains this > > entry_points = dict( > > console_scripts = [ > > 'rml2pdf=rlextra.rml2pdf.rml2pdf:main', > > > > 'pageCatcher=rlextra.pageCatcher.pageCatcher:s > > criptInterp', > > > > 'pdfexplorer=rlextra.pageCatcher.pdfexplorer:t > > est', > > ], > > gui_scripts = [ > > > > 'diagra=rlextra.graphics.guiedit.guiedit:mainA > > pp', > > ], > > ), > > this appears to work OK and creates all 4 scripts in Windows, Linux & Mac > OS, > however, when uninstalling or upgrading I get an error for Mac OS when > instead > of trying to remove a pageCatcher script it tries instead to remove > pagecatcher. [...] > > IOError: [Errno 2] No such file or directory: > > '/Users/rptlab/tmp/ttt/bin/pagecatcher' > > > > Storing debug log for failure in /Users/rptlab/.pip/pip.log > this is with pip 1.5.2 & python 2.7.6 in Darwin 10.8.0 and other later Macs. > Is this just a case issue? My guess it that it is just a case issue, since OS X file systems are often (but not always) case insensitive. Suggest you open an issue about it on the pip issue tracker: https://github.com/pypa/pip/issues -- Ned Deily, nad at acm.org From ncoghlan at gmail.com Wed Oct 15 09:44:05 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 15 Oct 2014 17:44:05 +1000 Subject: [Distutils] some questions about PEP470 In-Reply-To: References: <20141009164139.GE7954@merlinux.eu> <89513014-4E0E-422D-AD12-95730EED60A8@stufft.io> <20141011063148.GN7954@merlinux.eu> <264E2355-E619-476E-992B-8D11BBBFB217@stufft.io> <12C9C976-9B22-4D29-9CD9-C9CC66576D9D@stufft.io> <818DF4B5-FFAC-4F12-B0A0-3FF8970DF12D@stufft.io> Message-ID: On 15 October 2014 12:20, Stefan Krah wrote: > > > At this point (and possibly before) you are just trolling and not worth > any further correspondence. If some of your feigned surprise questions > are actually genuine, I recommend walking away from the keyboard for a > couple of weeks and reading some literature. > > Otherwise it is just a waste of time to attempt any further conversation. Stefan, I personally apologise if at any point you, or any other developer that chooses to use non-PyPI hosting, has ever been (or even felt) personally attacked over your hosting choices. There's no way we can hope to ensure that folks that are passionate about the end user experience offered by the Python packaging ecosystem first come up to speed on the subtleties of copyright licensing (and the concerns around the current clause in PyPI's conditions of use that ensures, amongst other things, the legal right to operate PyPI mirrors) or US export restrictions (and the fact that uploading to PyPI requires agreeing to abide by them, which may not be an acceptable condition for non-US based developers). However, balancing multiple competing viewpoints is *why* we have the Python Enhancement Proposal process, and this is why the accepted PEPs have always included external hosting support, even when the feature has been missing from some of the earlier draft proposals. You don't need to fight that battle any more - it was won a long time ago (and was never really in danger of being lost). At the same time, silently introducing additional points of failure into users' infrastructure, and especially introducing silent vulnerabilities to man-in-the-middle DNS hijacking attacks, is *not* OK. Eliminating those two problems has been the focus of many of the more controversial PyPI changes over the past year or more, so it is entirely understandable that developers that choose to use external hosting *have* felt personally attacked, especially when other developers have failed to understand why "just host your packages on PyPI" isn't always going to be an acceptable answer. The latest proposal in PEP 470 is designed to provide a mechanism which is completely explicit (so end users always know when they're depending on additional repositories beyond PyPI), while also relatively streamlined (so end users don't complain when a developer chooses to make use of the external hosting support). The previous design in PEP 438 ended up failing on both of those counts, which is why there is now this new proposal to replace it with a different mechanism that has been designed to address the existing usability challenges. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From holger at merlinux.eu Wed Oct 15 15:26:27 2014 From: holger at merlinux.eu (holger krekel) Date: Wed, 15 Oct 2014 13:26:27 +0000 Subject: [Distutils] devpi-server-2.1.1: replication/url fixes Message-ID: <20141015132627.GG7954@merlinux.eu> Hi all, the caching and private pypi server, devpi-server-2.1.1, is out and fixes some bugs, see changelog below. It is fully backward compatible, no export/import cycle required. For more info, see http://doc.devpi.net. best, holger krekel, merlinux GmbH 2.1.1 ---------------- - fix replication issue reported by a customer: if a replica lags behind a master and a file was created and then deleted meanwhile, the replica could get stuck with a FileReplicationError. We now let the master report a 410 GONE code so that the replica knows it can safely proceed because the file was deleted later anyways. - generate "systemd" configuration example when "--gen-config" is issued. Thanks Pavel Sedlak. - fix issue109: fix relative URLs in simple index pages and 404 errors on uploading toxresults and downloading files when serving under an outside URL with a sub path. Thanks to Joe Holloway for detailed infos. - drop limitation on maximum documentation size. Body size is now only controlled by frontends such as nginx. Thanks Stephan Erb. - use newer version of virtualenv for jenkins trigger. Thank brunsgaard. From richard at python.org Thu Oct 16 06:21:58 2014 From: richard at python.org (Richard Jones) Date: Thu, 16 Oct 2014 15:21:58 +1100 Subject: [Distutils] Process for taking over abandoned packages In-Reply-To: References: <20141014075430.GX7954@merlinux.eu> Message-ID: I am currently trying to decide what to do in the case of this request: https://sourceforge.net/p/pypi/support-requests/420/ It is remarkably similar to the request which only just recently got me into trouble, with the slight difference that there may be a trademark issue which I am definitely not able to address. Therefore I am extremely hesitant to actually grant the request. It certainly falls under the final statement in the written-up policy: "Transfer will not be performed where an individual or project wishes to take ownership of a name because they feel the current owner has left it stagnating, or even broken, in the absence of the above transfer conditions." Your thoughts, as always, are welcome. Richard -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Oct 16 07:47:09 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 16 Oct 2014 15:47:09 +1000 Subject: [Distutils] Process for taking over abandoned packages In-Reply-To: References: <20141014075430.GX7954@merlinux.eu> Message-ID: On 16 October 2014 14:21, Richard Jones wrote: > I am currently trying to decide what to do in the case of this request: > https://sourceforge.net/p/pypi/support-requests/420/ A couple of points on the specific request: 1. http://sourceforge.net/p/py-googlemaps/bugs/4/ indicates the package owner is aware their package doesn't work anymore, and is redirecting folks to pygeocoder. 2. As noted in the draft policy, the next step would be to file an issue at http://sourceforge.net/p/py-googlemaps/bugs/ to provide a public record and timeline of attempts to contact the existing owner (I know email is a pretty unreliable way of contacting me, for example, since I don't read all of mine - if Gmail flags it as promotional, social media, or spam, I may never see it) > It is remarkably similar to the request which only just recently got me into > trouble, with the slight difference that there may be a trademark issue > which I am definitely not able to address. Therefore I am extremely hesitant > to actually grant the request. It certainly falls under the final statement > in the written-up policy: > > "Transfer will not be performed where an individual or project wishes to > take ownership of a name because they feel the current owner has left it > stagnating, or even broken, in the absence of the above transfer > conditions." Moving on to more general comments on the draft policy, I'm inclined to agree with Holger's comment on that paragraph: if someone abandons a project entirely, but there's someone else willing and able to take it over, then it's desirable for us to have an agreed way to deal with that. His proposed thresholds of no releases for two years, and no response to a tracker issue regarding the transfer for 2 months sound reasonable to me, but I'd be quite open to bumping them higher (in the case at hand, the last py-googlemaps release and the last commit were both 5 years ago, while their last tracker activity was almost a year and a half ago). > Your thoughts, as always, are welcome. I see Noah already raised the trademark dispute question as a comment on the draft policy, but you may want to make it explicit that in the absence of an amicable resolution at the community level, trademark related questions would need to be escalated to the Python Software Foundation as the legal entity backing the PyPI service. I'd obviously prefer to see us resolve things through the more collaborative community processes, but talking to lawyers (both ours and other people's) is one of the reasons the PSF exists. At this stage, the point of escalation would be directly to the board - dealing with *other people's* trademarks is not something that is currently delegated to anyone. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ethan at stoneleaf.us Thu Oct 16 08:29:06 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 15 Oct 2014 23:29:06 -0700 Subject: [Distutils] Process for taking over abandoned packages In-Reply-To: References: <20141014075430.GX7954@merlinux.eu> Message-ID: <543F65B2.50207@stoneleaf.us> +1 to everything Nick said. -- ~Ethan~ From tleeuwenburg at gmail.com Thu Oct 16 06:41:31 2014 From: tleeuwenburg at gmail.com (Tennessee Leeuwenburg) Date: Thu, 16 Oct 2014 15:41:31 +1100 Subject: [Distutils] Process for taking over abandoned packages In-Reply-To: References: <20141014075430.GX7954@merlinux.eu> Message-ID: I think package owners have a responsibility to be contactable, but that the time-frames should be long to allow for the fact people do go off the grid for valid reasons sometimes. I'd suggest something like 2 months is a reasonable interval to allow; if contact via email and available social media fail after 2 months, then broken packages without evidence of maintenance in the source repository should be moveable. Inactive, abandoned and nonfunctional packages could be defined as: -- Owner uncontactable via email or social media for 2 months -- No alternative representative for the package identified -- No evidence of maintenance in the source repository for 1 year (?) -- Package is not installable via pip install on 2.7 or 3.x This is as distinct from the squatter clause, which is really about never-even-started, never-did-anything registrations. If the original owner wants to keep the name, then I think they get to keep it regardless of the requesting parties desire to get a hold of it. Onus should be on the requesting party to supply evidence of the criteria via the issue tracker. Making things too complex? Maybe... On 16 October 2014 15:21, Richard Jones wrote: > I am currently trying to decide what to do in the case of this request: > https://sourceforge.net/p/pypi/support-requests/420/ > > It is remarkably similar to the request which only just recently got me > into trouble, with the slight difference that there may be a trademark > issue which I am definitely not able to address. Therefore I am extremely > hesitant to actually grant the request. It certainly falls under the final > statement in the written-up policy: > > "Transfer will not be performed where an individual or project wishes to > take ownership of a name because they feel the current owner has left it > stagnating, or even broken, in the absence of the above transfer > conditions." > > Your thoughts, as always, are welcome. > > > Richard > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -- -------------------------------------------------- Tennessee Leeuwenburg http://myownhat.blogspot.com/ "Don't believe everything you think" -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefankrah at freenet.de Thu Oct 16 18:33:19 2014 From: stefankrah at freenet.de (Stefan Krah) Date: Thu, 16 Oct 2014 16:33:19 +0000 (UTC) Subject: [Distutils] some questions about PEP470 References: <20141009164139.GE7954@merlinux.eu> <89513014-4E0E-422D-AD12-95730EED60A8@stufft.io> <20141011063148.GN7954@merlinux.eu> <264E2355-E619-476E-992B-8D11BBBFB217@stufft.io> <12C9C976-9B22-4D29-9CD9-C9CC66576D9D@stufft.io> <818DF4B5-FFAC-4F12-B0A0-3FF8970DF12D@stufft.io> Message-ID: Nick Coghlan gmail.com> writes: > On 15 October 2014 12:20, Stefan Krah freenet.de> wrote: > > > > At this point (and possibly before) you are just trolling and not worth > > any further correspondence. If some of your feigned surprise questions > > are actually genuine, I recommend walking away from the keyboard for a > > couple of weeks and reading some literature. > > Otherwise it is just a waste of time to attempt any further conversation. [In case anyone is using a non-threaded view: The above paragraph was not directed at Nick.] > Stefan, I personally apologise if at any point you, or any other > developer that chooses to use non-PyPI hosting, has ever been (or even > felt) personally attacked over your hosting choices. I feel a bit embarrassed, since you have absolutely nothing to apologize for. It is a very nice gesture, so thanks! > There's no way we > can hope to ensure that folks that are passionate about the end user > experience offered by the Python packaging ecosystem first come up to > speed on the subtleties of copyright licensing (and the concerns > around the current clause in PyPI's conditions of use that ensures, > amongst other things, the legal right to operate PyPI mirrors) or US > export restrictions (and the fact that uploading to PyPI requires > agreeing to abide by them, which may not be an acceptable condition > for non-US based developers). Some people who *should* know better have repeatedly engaged in off-list FUD and personal attacks. It is nowhere near the level that, say, Lennart Poettering had to deal with, but enough for me to lose quite a bit of interest in Python core development. Stefan Krah From tseaver at palladion.com Thu Oct 16 23:47:14 2014 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 16 Oct 2014 17:47:14 -0400 Subject: [Distutils] some questions about PEP470 In-Reply-To: References: <20141009164139.GE7954@merlinux.eu> <89513014-4E0E-422D-AD12-95730EED60A8@stufft.io> <20141011063148.GN7954@merlinux.eu> <264E2355-E619-476E-992B-8D11BBBFB217@stufft.io> <12C9C976-9B22-4D29-9CD9-C9CC66576D9D@stufft.io> <818DF4B5-FFAC-4F12-B0A0-3FF8970DF12D@stufft.io> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/14/2014 09:15 PM, Donald Stufft wrote: > On Oct 14, 2014, at 8:50 PM, Stefan Krah > wrote: > >>> Donald Stufft stufft.io> writes: >>> >>>>> If you're this upset over someone redistributing your work, >>>>> then maybe Open Source Software is the wrong hobby for you. >>> >>> Usually one does not tell a core developer that his contributions >>> are "a hobby". I have contributed 40000+ lines of original, dense >>> C code, backed by 100% code coverage and 30000+ lines of ACL2 >>> proofs. > Uhh, maybe you?re misunderstanding the word hobby, unless you?re > getting paid for your OSS work you?re not doing it professionally. A > hobby isn?t a negative thing, until last December my OSS work was > entirely a hobby too, and it?s still a hobby in my spare time too. That use of "hobby" vs. "professional" is completely irrelevant to the discussion, and is effectively condescending and ad hominem. In my experience, the quality and committment of a developer's open source work are often unrelated to whether she gets paid directly by an employer to do it. Getting paid *does* make it possible to devote more time, rather than passion, to one's projects. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAlRAPOIACgkQ+gerLs4ltQ5vWACfRDCmT5yjkqfeBB+4xGAiBnAv n7MAoKag+7GkicRdZ9eSpJdz+HKml6aG =5Uxr -----END PGP SIGNATURE----- From graffatcolmingov at gmail.com Fri Oct 17 01:54:50 2014 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Thu, 16 Oct 2014 18:54:50 -0500 Subject: [Distutils] Hobby time (was: some questions about PEP470) In-Reply-To: References: Message-ID: On Oct 16, 2014 4:47 PM, "Tres Seaver" wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 10/14/2014 09:15 PM, Donald Stufft wrote: > > On Oct 14, 2014, at 8:50 PM, Stefan Krah > > wrote: > > > >>> Donald Stufft stufft.io> writes: > >>> > >>>>> If you're this upset over someone redistributing your work, > >>>>> then maybe Open Source Software is the wrong hobby for you. > >>> > >>> Usually one does not tell a core developer that his contributions > >>> are "a hobby". I have contributed 40000+ lines of original, dense > >>> C code, backed by 100% code coverage and 30000+ lines of ACL2 > >>> proofs. > > Uhh, maybe you?re misunderstanding the word hobby, unless you?re > > getting paid for your OSS work you?re not doing it professionally. A > > hobby isn?t a negative thing, until last December my OSS work was > > entirely a hobby too, and it?s still a hobby in my spare time too. > > That use of "hobby" vs. "professional" is completely irrelevant to the > discussion, and is effectively condescending and ad hominem. In my > experience, the quality and committment of a developer's open source work > are often unrelated to whether she gets paid directly by an employer to > do it. Getting paid *does* make it possible to devote more time, rather > than passion, to one's projects. > > > > Tres. > - -- > =================================================================== > Tres Seaver +1 540-429-0999 tseaver at palladion.com > Palladion Software "Excellence by Design" http://palladion.com > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.11 (GNU/Linux) > > iEYEARECAAYFAlRAPOIACgkQ+gerLs4ltQ5vWACfRDCmT5yjkqfeBB+4xGAiBnAv > n7MAoKag+7GkicRdZ9eSpJdz+HKml6aG > =5Uxr > -----END PGP SIGNATURE----- > A hobby is by definition what you do in your free time. Donald is clearly encouraging Stefan to spend his free time as he wishes. None of us who participate in our free time are obligated to support anything beyond our desire to do so, unlike someone who's paid to work on their project(s). It almost seems like you're nitpicking the valid usage of a word to pile into an argument in which you have nothing valid to say. I know it doesn't usually happen on mailing lists, but that's what it seems like here. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Oct 17 04:07:50 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 17 Oct 2014 12:07:50 +1000 Subject: [Distutils] Process for taking over abandoned packages In-Reply-To: References: <20141014075430.GX7954@merlinux.eu> Message-ID: On 16 October 2014 14:41, Tennessee Leeuwenburg wrote: > I think package owners have a responsibility to be contactable, but that the > time-frames should be long to allow for the fact people do go off the grid > for valid reasons sometimes. I'd suggest something like 2 months is a > reasonable interval to allow; if contact via email and available social > media fail after 2 months, then broken packages without evidence of > maintenance in the source repository should be moveable. > > Inactive, abandoned and nonfunctional packages could be defined as: > -- Owner uncontactable via email or social media for 2 months The project issue tracker (if it has one) should be included here - the problem with email and social media is that they will generally lack the clear audit trail that public issue trackers automatically provide. > -- Package is not installable via pip install on 2.7 or 3.x I'd be very wary of including technical requirements like this in the package transfer process. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Fri Oct 17 07:00:32 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 17 Oct 2014 15:00:32 +1000 Subject: [Distutils] Hobby time (was: some questions about PEP470) In-Reply-To: References: Message-ID: On 17 October 2014 09:54, Ian Cordasco wrote: > > A hobby is by definition what you do in your free time. Donald is clearly > encouraging Stefan to spend his free time as he wishes. None of us who > participate in our free time are obligated to support anything beyond our > desire to do so, unlike someone who's paid to work on their project(s). It > almost seems like you're nitpicking the valid usage of a word to pile into > an argument in which you have nothing valid to say. I know it doesn't > usually happen on mailing lists, but that's what it seems like here. Folks, when somebody tells you that they find a particular term diminutive and insulting, *it is not OK* to tell them that they shouldn't feel that way. An honest description of how someone else's choice of words makes us feel should *never* be casually dismissed. In this particular case, not everything people do in their free time is appropriately classified as a hobby. Volunteering for charities, serving on the boards of corporations and non-profit organisations, and, in many cases, contributing to the open source community is better characterised as "volunteer work". A lawyer representing someone pro bono is expected to be just as professional as they would if they were getting paid, and the same is true for most open source contributors that identify as professional software developers. Yes, we're contributing on our own time on the community side of things, but we're bringing just as much professional expertise to bear regardless of whether we're getting paid or not. The fact is that community members (including Stefan) *have* been personally attacked, with examples including accusations of not caring about their users or the Python community as whole, due to their concerns around some of the legal details related to hosting packages directly on PyPI. In particular, PyPI is subject to US export regulations, a legal entanglement that developers from outside the US may prefer to avoid, and there's a clause in the terms and conditions which introduces some ambiguity as to whether or not it is attempting to override a project's own stated license (this isn't the intention, but the clause can certainly be read that way). In relation to the latter point, Red Hat, *as an organisation*, has proposed alternate wording for that clause (to no avail thus far). Finding time to pursue that kind of legally sensitive change is always a challenge (as it only affects a relatively small subset of package authors, and has potentially broad ramifications for the rights of PyPI consumers), but I expect the topic to eventually be raised again via the Python-legal-sig mailing list. These are not concerns that can just be waved away - they need to be taken seriously, and I believe both PEP 438 and PEP 470 represent good faith attempts to do just that. While it *is* convenient for end users when developers choose to host their releases directly on PyPI, that benefit doesn't override the interests of developers in choosing what licenses they choose to place on their work, or which legal regimes they choose to grant authority to. Offering an explicit, reliable, user experience for externally hosted packages is key to simultaneously respecting the interests of both end users downloading and discovering packages via PyPI, and developers retaining autonomy in relation to how they choose to engage with the intricacies of the global copyright system. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From kelly.goedert at gmail.com Wed Oct 22 21:00:52 2014 From: kelly.goedert at gmail.com (Kelly Goedert) Date: Wed, 22 Oct 2014 17:00:52 -0200 Subject: [Distutils] stdeb and python dependencies Message-ID: Hi, I am new to python. I created a simple python cli application that depends on jsonpickle and plumbum. Using this command python setup.py --command-packages=stdeb.command bdist_deb I was able to create a .deb package. But when I try to install the package with dpkg -i package_name I get package_name depends on python-plumbum; however: Package python-plumbum is not installed. package_name depends on python-jsonpickle; however: Package python-jsonpickle In my setup.py I have this: dependencies=['plumbum','jsonpickle'] install_requires=dependencies Is it possible to make the debian package install this python dependencies? Or is this the wrong approach? If so, what would be the recommend way? Thanks Kelly -------------- next part -------------- An HTML attachment was scrubbed... URL: From rupert.kolb at uni-tuebingen.de Thu Oct 23 17:01:50 2014 From: rupert.kolb at uni-tuebingen.de (Rupert Kolb) Date: Thu, 23 Oct 2014 17:01:50 +0200 Subject: [Distutils] Where is distribute at pypi? Message-ID: <5449185E.8040509@uni-tuebingen.de> Hi, where is the distribute package at pypi? There is software (for instance pydicom) which need the distribute-0.6.xxx package. It doesn't work with the "setuptools" replacement out of the box! You can not just remove it!? Rupert From barry at python.org Thu Oct 23 19:50:57 2014 From: barry at python.org (Barry Warsaw) Date: Thu, 23 Oct 2014 13:50:57 -0400 Subject: [Distutils] stdeb and python dependencies References: Message-ID: <20141023135057.6348a039@anarchist> On Oct 22, 2014, at 05:00 PM, Kelly Goedert wrote: >I am new to python. I created a simple python cli application that depends >on jsonpickle and plumbum. > >Using this command python setup.py --command-packages=stdeb.command >bdist_deb I was able to create a .deb package. But when I try to install >the package with dpkg -i package_name I get > > package_name depends on python-plumbum; however: > Package python-plumbum is not installed. > package_name depends on python-jsonpickle; however: > Package python-jsonpickle > >In my setup.py I have this: > >dependencies=['plumbum','jsonpickle'] >install_requires=dependencies > >Is it possible to make the debian package install this python dependencies? >Or is this the wrong approach? If so, what would be the recommend way? `dpkg -i` doesn't install dependencies. After you get this, run `sudo apt-get install -f` to get your missing dependencies. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: not available URL: From erik.m.bray at gmail.com Thu Oct 23 21:31:32 2014 From: erik.m.bray at gmail.com (Erik Bray) Date: Thu, 23 Oct 2014 15:31:32 -0400 Subject: [Distutils] Where is distribute at pypi? In-Reply-To: <5449185E.8040509@uni-tuebingen.de> References: <5449185E.8040509@uni-tuebingen.de> Message-ID: On Thu, Oct 23, 2014 at 11:01 AM, Rupert Kolb wrote: > Hi, > > where is the distribute package at pypi? > There is software (for instance pydicom) which need the > distribute-0.6.xxx package. It doesn't work with the "setuptools" > replacement out of the box! > You can not just remove it!? Well distribute is right here: https://pypi.python.org/pypi/distribute/0.7.3 But nothing should be explicitly depending on it (particularly in favor of setuptools). Looking at pydicom there's nothing remarkable about it that it should make such a requirement, so if they do they're wrong and its developers should fix that. From dholth at gmail.com Thu Oct 23 22:10:23 2014 From: dholth at gmail.com (Daniel Holth) Date: Thu, 23 Oct 2014 16:10:23 -0400 Subject: [Distutils] Where is distribute at pypi? In-Reply-To: References: <5449185E.8040509@uni-tuebingen.de> Message-ID: There's your problem. https://code.google.com/p/pydicom/source/browse/source/setup.py . It's trying to install distribute as a side effect of running setup.py, so it can import setuptools. This is now discouraged. The best solution would be to remove the first 3 lines of that file. Setuptools is guaranteed to be available when installing with pip. From me at m.merickel.org Thu Oct 23 22:23:28 2014 From: me at m.merickel.org (Michael Merickel) Date: Thu, 23 Oct 2014 15:23:28 -0500 Subject: [Distutils] depending on setuptools is discouraged? Message-ID: I'm noticing a trend that depending on setuptools is discouraged[1] in the install_requires of your setup.py. However, some packages like pyramid have core features that depend on pkg_resources (which is part of setuptools). Thus, we depend on it. As I've monitored distutils-sig I haven't seen any talk of separating these packages. To me, requiring the use of pkg_resources is a very valid use-case for depending on setuptools as it provides many improvements over the basic `pkgutil.get_data` API. Since I'm not aware of a workaround for this problem, and I'm seeing setuptools be categorically discouraged, perhaps I am just unaware and need to be enlightened? Thanks! [1] https://mail.python.org/pipermail/distutils-sig/2014-October/025131.html From donald at stufft.io Thu Oct 23 22:38:56 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 23 Oct 2014 16:38:56 -0400 Subject: [Distutils] depending on setuptools is discouraged? In-Reply-To: References: Message-ID: <46F548F7-7036-494E-9030-57A812D026D2@stufft.io> > On Oct 23, 2014, at 4:23 PM, Michael Merickel wrote: > > I'm noticing a trend that depending on setuptools is discouraged[1] in > the install_requires of your setup.py. However, some packages like > pyramid have core features that depend on pkg_resources (which is part > of setuptools). Thus, we depend on it. As I've monitored distutils-sig > I haven't seen any talk of separating these packages. To me, requiring > the use of pkg_resources is a very valid use-case for depending on > setuptools as it provides many improvements over the basic > `pkgutil.get_data` API. > > Since I'm not aware of a workaround for this problem, and I'm seeing > setuptools be categorically discouraged, perhaps I am just unaware and > need to be enlightened? > > Thanks! > > [1] https://mail.python.org/pipermail/distutils-sig/2014-October/025131.html > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig To be specific that post was talking about using the idiom that will automatically bootstrap setuptools when you run setup.py:: from distribute_setup import use_setuptools use_setuptools(version="0.6.49?) Projects *should* depend on setuptools in install_requires if their project itself uses something inside of setuptools as a runtime dependency (e.g. not in setup.py). --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From doko at ubuntu.com Thu Oct 23 22:47:33 2014 From: doko at ubuntu.com (Matthias Klose) Date: Thu, 23 Oct 2014 22:47:33 +0200 Subject: [Distutils] depending on setuptools is discouraged? In-Reply-To: <46F548F7-7036-494E-9030-57A812D026D2@stufft.io> References: <46F548F7-7036-494E-9030-57A812D026D2@stufft.io> Message-ID: <54496965.6080703@ubuntu.com> Am 23.10.2014 um 22:38 schrieb Donald Stufft: > >> On Oct 23, 2014, at 4:23 PM, Michael Merickel wrote: >> >> I'm noticing a trend that depending on setuptools is discouraged[1] in >> the install_requires of your setup.py. However, some packages like >> pyramid have core features that depend on pkg_resources (which is part >> of setuptools). Thus, we depend on it. As I've monitored distutils-sig >> I haven't seen any talk of separating these packages. To me, requiring >> the use of pkg_resources is a very valid use-case for depending on >> setuptools as it provides many improvements over the basic >> `pkgutil.get_data` API. >> >> Since I'm not aware of a workaround for this problem, and I'm seeing >> setuptools be categorically discouraged, perhaps I am just unaware and >> need to be enlightened? >> >> Thanks! >> >> [1] https://mail.python.org/pipermail/distutils-sig/2014-October/025131.html >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig > > To be specific that post was talking about using the idiom that will > automatically bootstrap setuptools when you run setup.py:: > > from distribute_setup import use_setuptools > use_setuptools(version="0.6.49?) > > Projects *should* depend on setuptools in install_requires if their > project itself uses something inside of setuptools as a runtime > dependency (e.g. not in setup.py). I don't know about many packages needing setuptools as a runtime dependency. Most packages need a dependency on pkg_resources only. So adding a pkg-resources egg would be nice. From donald at stufft.io Thu Oct 23 22:50:02 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 23 Oct 2014 16:50:02 -0400 Subject: [Distutils] depending on setuptools is discouraged? In-Reply-To: <54496965.6080703@ubuntu.com> References: <46F548F7-7036-494E-9030-57A812D026D2@stufft.io> <54496965.6080703@ubuntu.com> Message-ID: <0B284153-B9C7-4083-9632-C40649C0B548@stufft.io> > > On Oct 23, 2014, at 4:47 PM, Matthias Klose wrote: > > Am 23.10.2014 um 22:38 schrieb Donald Stufft: >> >>> On Oct 23, 2014, at 4:23 PM, Michael Merickel wrote: >>> >>> I'm noticing a trend that depending on setuptools is discouraged[1] in >>> the install_requires of your setup.py. However, some packages like >>> pyramid have core features that depend on pkg_resources (which is part >>> of setuptools). Thus, we depend on it. As I've monitored distutils-sig >>> I haven't seen any talk of separating these packages. To me, requiring >>> the use of pkg_resources is a very valid use-case for depending on >>> setuptools as it provides many improvements over the basic >>> `pkgutil.get_data` API. >>> >>> Since I'm not aware of a workaround for this problem, and I'm seeing >>> setuptools be categorically discouraged, perhaps I am just unaware and >>> need to be enlightened? >>> >>> Thanks! >>> >>> [1] https://mail.python.org/pipermail/distutils-sig/2014-October/025131.html >>> _______________________________________________ >>> Distutils-SIG maillist - Distutils-SIG at python.org >>> https://mail.python.org/mailman/listinfo/distutils-sig >> >> To be specific that post was talking about using the idiom that will >> automatically bootstrap setuptools when you run setup.py:: >> >> from distribute_setup import use_setuptools >> use_setuptools(version="0.6.49?) >> >> Projects *should* depend on setuptools in install_requires if their >> project itself uses something inside of setuptools as a runtime >> dependency (e.g. not in setup.py). > > I don't know about many packages needing setuptools as a runtime dependency. > Most packages need a dependency on pkg_resources only. So adding a pkg-resources > egg would be nice. As oh right now pkg_resources is part of setuptools, it?s only the downstream folks who split them apart. There are some decisions to make regarding splitting them out that I don?t know how setuptools wants to handle it. However once the stuff that pkg_resources provides is standardized it will be available as part of https://warehouse.python.org/project/packaging/ so projects wishing to use the standards based functionality (most of which is heavily inspired if not outright ?stolen? from setuptools/pkg_resources) will be able to use that eventually. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From me at m.merickel.org Thu Oct 23 22:57:08 2014 From: me at m.merickel.org (Michael Merickel) Date: Thu, 23 Oct 2014 15:57:08 -0500 Subject: [Distutils] depending on setuptools is discouraged? In-Reply-To: <0B284153-B9C7-4083-9632-C40649C0B548@stufft.io> References: <46F548F7-7036-494E-9030-57A812D026D2@stufft.io> <54496965.6080703@ubuntu.com> <0B284153-B9C7-4083-9632-C40649C0B548@stufft.io> Message-ID: On Thu, Oct 23, 2014 at 3:50 PM, Donald Stufft wrote: > As oh right now pkg_resources is part of setuptools, it?s only the downstream > folks who split them apart. There are some decisions to make regarding splitting > them out that I don?t know how setuptools wants to handle it. > > However once the stuff that pkg_resources provides is standardized it will be > available as part of https://warehouse.python.org/project/packaging/ so projects > wishing to use the standards based functionality (most of which is heavily inspired > if not outright ?stolen? from setuptools/pkg_resources) will be able to use that > eventually. It's great to hear that some work is being done on splitting them out, thanks for the response! From greg.ewing at canterbury.ac.nz Thu Oct 23 23:07:36 2014 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 24 Oct 2014 10:07:36 +1300 Subject: [Distutils] depending on setuptools is discouraged? In-Reply-To: <54496965.6080703@ubuntu.com> References: <46F548F7-7036-494E-9030-57A812D026D2@stufft.io> <54496965.6080703@ubuntu.com> Message-ID: <54496E18.5020208@canterbury.ac.nz> Matthias Klose wrote: > Most packages need a dependency on pkg_resources only. So adding a pkg-resources > egg would be nice. I thought that making pkg_resources a separate thing was one of the goals of the new distutils regime. Has it not happened yet? -- Greg From holger at merlinux.eu Fri Oct 24 17:36:34 2014 From: holger at merlinux.eu (holger krekel) Date: Fri, 24 Oct 2014 15:36:34 +0000 Subject: [Distutils] devpi-{server-2.1.2, web-2.2.1}: bugfixes, streamlined web templates Message-ID: <20141024153634.GN22258@merlinux.eu> devpi-server-2.1.2 and devpi-web-2.2.1 bring a host of fixes to the private pypi server system. You can upgrade without migrating your data if you run already with devpi-server-2.1.X. Find docs as usual at: http://doc.devpi.net Many thanks to Florian Schulze who did most of the changes in devpi-web. Have fun, holger krekel, merlinux GmbH devpi-server-2.1.2 ------------------ - fix issue172: avoid traceback when user/index/name/version is accessed. - fix issue170: ensure that we parse the prospective pip-6.0 user agent string properly so that using the username/index url works with pip. Thanks Donald Stufft and Florian Schulze. - fix issue158: redirect to normalized projectname for all GET views. - fix issue169: change /+status to expose "event_serial" as "the last event serial that was processed". document "serial" and "event-serial" and also refine internals wrt to "event-serial" so that it means the "last serial for which events have been processed" devpi-web-2.2.1 --------------- - require devpi-server>=2.1.2 - fix issue175: use normalized name of projects, so redirects from unnormalized names works. NOTE that if you had issues with documentation uploads not appearing because of normalization issues ("-" or "_" appearing in the name for example) you need to re-upload the docs or do a full export/import cycle. - fix view when tox results can not be parsed. - version.pt: removed "code" tag around overwrite count. - macros.pt: added "footer" tag around the whole footer part. - version.pt: moved file type, python version and size info from their own columns into the file column. - version.pt: moved history column from before the tox results column to behind the tox results. - version.pt: removed "last modified" from history column - version.pt: removed timestamp from "replaced" action in history column - version.pt: add link to PyPI page if applicable. - fix project page view if there are downloads with filenames which can't be parsed as packages with version number - fix notfound-redirect when serving under an outside URL with a sub path From ncoghlan at gmail.com Sat Oct 25 15:17:06 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 25 Oct 2014 23:17:06 +1000 Subject: [Distutils] depending on setuptools is discouraged? In-Reply-To: <54496E18.5020208@canterbury.ac.nz> References: <46F548F7-7036-494E-9030-57A812D026D2@stufft.io> <54496965.6080703@ubuntu.com> <54496E18.5020208@canterbury.ac.nz> Message-ID: On 24 October 2014 07:07, Greg Ewing wrote: > Matthias Klose wrote: >> >> Most packages need a dependency on pkg_resources only. So adding a >> pkg-resources >> egg would be nice. > > I thought that making pkg_resources a separate thing was one > of the goals of the new distutils regime. Has it not happened > yet? My understanding is that setuptools currently faces its own bootstrapping problems if pkg_resources is separated out. The accepted PEP 477 will backport ensurepip to Python 2.7.9+, at which point "python -m ensurepip && python -m pip install --upgrade setuptools" should reliably cover the bootstrapping problem even with pkg_resources separated out. That said, there also isn't a current setuptools issue to split them, so I don't know where this currently sits on Jason's priority list (setuptools itself really isn't that big a deal as a runtime dependency - the thing you don't want on your production servers is the compilers that setuptools needs in order to do anything useful with extension module source files). Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ethan at stoneleaf.us Mon Oct 27 14:47:45 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 27 Oct 2014 06:47:45 -0700 Subject: [Distutils] a package with is a module Message-ID: <544E4D01.3030900@stoneleaf.us> If there is an answer, tutorial, readme, or docs that would help, a link would be greatly appreciated. I have finally converted my dbf library to Python3, but I want to also continue supporting at least 2.7, and I see no reason to remove the existing library which also supports back to 2.4. I've never been very happy with the egg format (cool name, but having the source zipped up is irritating when one wants to look at it) and so I have left this library as a couple modules: dbf.py, and dbf_test.py. So I have multiple problems: - how do I tell PyPI that this file is for Python2.4 - 2.6, this other file is for 2.7, and this other other file is for 3.3+ ? - if I have to stick it all in one archive, how do I tell setup.py which to install? - is it possible to have all three source files as, say, .txt files, and then have some Python code before the setup() call which renames the correct one to dbf.py? How do I know where the .txt files are at to rename them? I have skimmed http://python-packaging-user-guide.readthedocs.org/en/latest/, but didn't find what I was looking for there. Thanks for any help. -- ~Ethan~ From ethan at stoneleaf.us Mon Oct 27 14:51:45 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 27 Oct 2014 06:51:45 -0700 Subject: [Distutils] a package with is a module In-Reply-To: <544E4D01.3030900@stoneleaf.us> References: <544E4D01.3030900@stoneleaf.us> Message-ID: <544E4DF1.2040207@stoneleaf.us> Argh. That should have been 'which is a module'. -- ~Ethan~ From p.f.moore at gmail.com Mon Oct 27 15:04:38 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 27 Oct 2014 14:04:38 +0000 Subject: [Distutils] a package with is a module In-Reply-To: <544E4D01.3030900@stoneleaf.us> References: <544E4D01.3030900@stoneleaf.us> Message-ID: On 27 October 2014 13:47, Ethan Furman wrote: > So I have multiple problems: > > - how do I tell PyPI that this file is for Python2.4 - 2.6, this other > file is for 2.7, and this other other file is for 3.3+ ? > > - if I have to stick it all in one archive, how do I tell setup.py which > to install? > > - is it possible to have all three source files as, say, .txt files, and > then have some Python code before the setup() call which renames the correct > one to dbf.py? How do I know where the .txt files are at to rename them? For a source distribution, you could play clever games in setup.py to put the right file in place, with the right name. But that's messy and it means that if you distribute wheels (not that there's much point in doing so) you need separate wheels for 2.6-, 2.7 and 3.3+. Alternatively, you could distribute all 3 files, as dbf \ - __init__.py - dbf_26.py - dbf_27.py - dbf_3.py Then in __init__.py do if sys.version_info[0] == 3: from .dbf_3 import * elif sys.version_info[:2] == (2, 7): from .dbf_27 import * else from .dbf_26 import * Paul From ethan at stoneleaf.us Mon Oct 27 16:07:43 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 27 Oct 2014 08:07:43 -0700 Subject: [Distutils] a package with is a module In-Reply-To: References: <544E4D01.3030900@stoneleaf.us> Message-ID: <544E5FBF.9040406@stoneleaf.us> On 10/27/2014 07:04 AM, Paul Moore wrote: > On 27 October 2014 13:47, Ethan Furman wrote: >> So I have multiple problems: >> >> - how do I tell PyPI that this file is for Python2.4 - 2.6, this other >> file is for 2.7, and this other other file is for 3.3+ ? >> >> - if I have to stick it all in one archive, how do I tell setup.py which >> to install? >> >> - is it possible to have all three source files as, say, .txt files, and >> then have some Python code before the setup() call which renames the correct >> one to dbf.py? How do I know where the .txt files are at to rename them? > > For a source distribution, you could play clever games in setup.py to > put the right file in place, with the right name. But that's messy and > it means that if you distribute wheels (not that there's much point in > doing so) you need separate wheels for 2.6-, 2.7 and 3.3+. But how? When setup.py runs, is it guaranteed to do so in a particular location relative to the installable files? And I wouldn't mind making separate wheels, if I ever get that far. > Alternatively, you could distribute all 3 files, as > > dbf > \ > - __init__.py > - dbf_26.py > - dbf_27.py > - dbf_3.py > > Then in __init__.py do > > if sys.version_info[0] == 3: > from .dbf_3 import * > elif sys.version_info[:2] == (2, 7): > from .dbf_27 import * > else > from .dbf_26 import * The problem I have with this method is that the last time I tried it setup.py attempted to pre-compile all the files, resulting in syntax errors, which makes for a lousy user experience. -- ~Ethan~ From p.f.moore at gmail.com Mon Oct 27 16:26:20 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 27 Oct 2014 15:26:20 +0000 Subject: [Distutils] a package with is a module In-Reply-To: <544E5FBF.9040406@stoneleaf.us> References: <544E4D01.3030900@stoneleaf.us> <544E5FBF.9040406@stoneleaf.us> Message-ID: On 27 October 2014 15:07, Ethan Furman wrote: >> For a source distribution, you could play clever games in setup.py to >> put the right file in place, with the right name. But that's messy and >> it means that if you distribute wheels (not that there's much point in >> doing so) you need separate wheels for 2.6-, 2.7 and 3.3+. > > But how? When setup.py runs, is it guaranteed to do so in a particular > location relative to the installable files? I don't know to be honest. It's not guaranteed to be run from a particular location (you can do python /whatever/setup.py if you want) but most setup.py scripts probably assume you're in the same directory as setup.py, so it's probably always going to be there for any practical purpose. I'd do something like (untested!) here = os.path.dirname(os.path.abspath(__file__)) if sys.version_info[0] == 3: src = 'dbf_3.py' elif sys.version_info[:2] == (2, 7): src = 'dbf_27.py' else src = 'dbf_26.py' shutil.copyfile(os.path.join(here, src), os.path.join(here, 'dbf.py')) setup ( ... py_modules=['dbf'], ... ) > And I wouldn't mind making separate wheels, if I ever get that far. OK, although you'd need to be careful to get the tags right. >> Alternatively, you could distribute all 3 files, as [...] > > The problem I have with this method is that the last time I tried it > setup.py attempted to pre-compile all the files, resulting in syntax errors, > which makes for a lousy user experience. Yeah, it's harmless but ugly. I've seen that issue myself. Paul From pmiscml at gmail.com Mon Oct 27 16:15:12 2014 From: pmiscml at gmail.com (Paul Sokolovsky) Date: Mon, 27 Oct 2014 17:15:12 +0200 Subject: [Distutils] Efficiently supporting multiple platforms in distribution packages Message-ID: <20141027171512.339f3656@x230> Hello, I just saw another topic posted, https://mail.python.org/pipermail/distutils-sig/2014-October/025140.html , which raises question similar to which I had in mind for some time. To not hijack that thread, I open a new one, but comment to a message from that thread, https://mail.python.org/pipermail/distutils-sig/2014-October/025142.html So, my case: support different platforms in one distribution package. To give more concrete example, let it be some module implemented using ctypes, with completely different implementation for Linux, MacOSX, and Windows. I'd also like to avoid installing files unneeded for particular platform (a usecase applies to MicroPython http://micropython.org/ , where there can be simply not enough storage space to install cruft). On Mon, 27 Oct 2014 14:04:38 +0000 Paul Moore wrote: > For a source distribution, you could play clever games in setup.py to > put the right file in place, with the right name. But that's messy and > it means that if you distribute wheels (not that there's much point in > doing so) you need separate wheels for 2.6-, 2.7 and 3.3+. Ok, so are there guidelines, best practices or at least example(s) how to do that? I pretty much would like to avoid inventing my own "clever games" to achieve that. > Alternatively, you could distribute all 3 files, as > > dbf > \ > - __init__.py > - dbf_26.py > - dbf_27.py > - dbf_3.py > > Then in __init__.py do > > if sys.version_info[0] == 3: > from .dbf_3 import * > elif sys.version_info[:2] == (2, 7): > from .dbf_27 import * > else > from .dbf_26 import * For our MicroPython case, we would like to avoid this due to aforementioned reasons. We could probably post-process installation dir after pip to remove unneeded files, but that sounds like hack - we'd rather do it fully on distribution package level, and target to install just a single source file per module, and avoid dispatcher like above (again, for efficiency reasons). There's also another issue - of supporting "cross-installs". Not all MicroPython targets support running pip natively, so it instead runs on a host computer, but would need be instructed to select source variants for a particular target platform. Any hints/pointers on how to achieve this - preferrably in "standard" way - are appreciated! -- Best regards, Paul mailto:pmiscml at gmail.com From marius at pov.lt Mon Oct 27 16:30:03 2014 From: marius at pov.lt (Marius Gedminas) Date: Mon, 27 Oct 2014 17:30:03 +0200 Subject: [Distutils] a package with is a module In-Reply-To: <544E4D01.3030900@stoneleaf.us> References: <544E4D01.3030900@stoneleaf.us> Message-ID: <20141027153003.GA8729@fridge.pov.lt> On Mon, Oct 27, 2014 at 06:47:45AM -0700, Ethan Furman wrote: > If there is an answer, tutorial, readme, or docs that would help, a > link would be greatly appreciated. > > I have finally converted my dbf library to Python3, but I want to > also continue supporting at least 2.7, and I see no reason to remove > the existing library which also supports back to 2.4. My objgraph is a single-file Python module that supports Pythons from 2.4 to 3.4. [*] [*] Well, it _probably_ supports 2.4 and 2.5; I don't actually test that because tox no longer supports 2.4 or 2.5 (and tox dropped support for those Pythons because virtualenv dropped support for them, IIRC). > So I have multiple problems: > > - how do I tell PyPI that this file is for Python2.4 - 2.6, this > other file is for 2.7, and this other other file is for 3.3+ ? Maintaining three separate files is a pain. Most people who support multiple Python versions would recommend sticking to a single source written in the common language subset. It's not that hard. http://python3porting.com/noconv.html has helpful suggestions. If you don't want to rely on the `six` module from PyPI, you can always copy the tricks it performs and do them yourself. > - if I have to stick it all in one archive, how do I tell setup.py which to install? Even if that's possible, I don't think it's a good idea. > - is it possible to have all three source files as, say, .txt > files, and then have some Python code before the setup() call which > renames the correct one to dbf.py? If you go this route I suggest copying instead of renaming. > How do I know where the .txt files are at to rename them? Many setup.py files fail if you run them when your working directory is not the one where the setup.py resides itself. I think you can safely rely on that implementation detail. Marius Gedminas -- If the facts don't fit the theory, change the facts. -- Albert Einstein -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: Digital signature URL: From p.f.moore at gmail.com Mon Oct 27 16:49:21 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 27 Oct 2014 15:49:21 +0000 Subject: [Distutils] Efficiently supporting multiple platforms in distribution packages In-Reply-To: <20141027171512.339f3656@x230> References: <20141027171512.339f3656@x230> Message-ID: On 27 October 2014 15:15, Paul Sokolovsky wrote: >> For a source distribution, you could play clever games in setup.py to >> put the right file in place, with the right name. But that's messy and >> it means that if you distribute wheels (not that there's much point in >> doing so) you need separate wheels for 2.6-, 2.7 and 3.3+. > > Ok, so are there guidelines, best practices or at least example(s) how > to do that? I pretty much would like to avoid inventing my own "clever > games" to achieve that. Personally, my guideline would be "don't do that". We're trying to move away from having complex code in setup.py, to a more nearly declarative solution. The above approach pretty much directly goes against that. I posted code later in the same thread, but I still don't think people should be doing it. I see the issue you and Ethan have, though. It should be considered in the context of Metadata 2.0 and things like that, so we make sure the use case is covered. Paul From dholth at gmail.com Mon Oct 27 18:24:22 2014 From: dholth at gmail.com (Daniel Holth) Date: Mon, 27 Oct 2014 13:24:22 -0400 Subject: [Distutils] Efficiently supporting multiple platforms in distribution packages In-Reply-To: References: <20141027171512.339f3656@x230> Message-ID: The closest thing we have right now is 2to3. It produces different installed code depending on how setup.py was run, and if you produce wheels they have a tag to distinguish them from each other based on the Python version. It's not wrong to have a complicated build process in setup.py. The confusion is that setup.py both builds the package and generates its metadata. It's common for the list of dependencies in the metadata to change based on the version of Python. Instead, we would prefer that a single package has the same metadata independent of how it is built. The common different-dependencies-per-Python-version-or-OS case is supported with "environment markers". They can be used to turn dependencies on or off with simple expressions. The wheel project's own setup.py uses them. So my strategy would be to use a single source package to generate several wheels tagged per Python implementation or version. Later the installer would be able to pick the correct one based on its tags. As ever beware of trying to extend distutils. It's not very good at that. From dholth at gmail.com Mon Oct 27 20:16:40 2014 From: dholth at gmail.com (Daniel Holth) Date: Mon, 27 Oct 2014 15:16:40 -0400 Subject: [Distutils] toml Message-ID: Learned about toml the other day, looks like it could be useful for packaging. It is an .ini-style language that maps unambiguously to a dict, supports comments, and is used by rust's packaging system "cargo". https://github.com/toml-lang/toml Unfortunately at least 3 of the Python implementations don't seem to be up to date with the current version of the spec specifically regarding triple-""" strings. From donald at stufft.io Mon Oct 27 20:23:28 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 27 Oct 2014 15:23:28 -0400 Subject: [Distutils] toml In-Reply-To: References: Message-ID: <33684B52-DC82-4DD9-9160-948C0B0F796F@stufft.io> > On Oct 27, 2014, at 3:16 PM, Daniel Holth wrote: > > Learned about toml the other day, looks like it could be useful for > packaging. It is an .ini-style language that maps unambiguously to a > dict, supports comments, and is used by rust's packaging system > "cargo". https://github.com/toml-lang/toml > > Unfortunately at least 3 of the Python implementations don't seem to > be up to date with the current version of the spec specifically > regarding triple-""" strings. > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig Ugh, I hate TOML. I?m -1 on any of the standards using it, but I also think the standards should be around data exchange and should just use JSON and leave front end stuff like that up to the implementations. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From p.f.moore at gmail.com Mon Oct 27 21:23:02 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 27 Oct 2014 20:23:02 +0000 Subject: [Distutils] toml In-Reply-To: <33684B52-DC82-4DD9-9160-948C0B0F796F@stufft.io> References: <33684B52-DC82-4DD9-9160-948C0B0F796F@stufft.io> Message-ID: On 27 October 2014 19:23, Donald Stufft wrote: > Ugh, I hate TOML. I?m -1 on any of the standards using it, but I also > think the standards should be around data exchange and should just use > JSON and leave front end stuff like that up to the implementations. I had a quick glance at TOML, and I can't say I was particularly enamoured by it. I don't see that it has any particularly huge benefits over "plain" ini files (if your needs are simple) or YAML (ignoring the over-complicated stuff that nobody actually needs). +1 on JSON for "internal" format, and tools deciding for themselves on the best user-facing format. I'm also not sure I see the value of mapping directly to a dict. Surely internal formats should be isolated from the user interface, not exposed directly? Paul From dholth at gmail.com Mon Oct 27 21:45:12 2014 From: dholth at gmail.com (Daniel Holth) Date: Mon, 27 Oct 2014 16:45:12 -0400 Subject: [Distutils] toml In-Reply-To: References: <33684B52-DC82-4DD9-9160-948C0B0F796F@stufft.io> Message-ID: I liked it because I agree with the TOML author that the YAML spec gives rage; YAML seems to be defined as a bunch of things that the end user is supposed to think are intuitive, but try understanding and correctly parsing the full set of what is allowed... TOML on the other hand is short. On Mon, Oct 27, 2014 at 4:23 PM, Paul Moore wrote: > On 27 October 2014 19:23, Donald Stufft wrote: >> Ugh, I hate TOML. I?m -1 on any of the standards using it, but I also >> think the standards should be around data exchange and should just use >> JSON and leave front end stuff like that up to the implementations. > > I had a quick glance at TOML, and I can't say I was particularly > enamoured by it. I don't see that it has any particularly huge > benefits over "plain" ini files (if your needs are simple) or YAML > (ignoring the over-complicated stuff that nobody actually needs). > > +1 on JSON for "internal" format, and tools deciding for themselves on > the best user-facing format. > > I'm also not sure I see the value of mapping directly to a dict. > Surely internal formats should be isolated from the user interface, > not exposed directly? > Paul From donald at stufft.io Mon Oct 27 21:58:19 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 27 Oct 2014 16:58:19 -0400 Subject: [Distutils] toml In-Reply-To: References: <33684B52-DC82-4DD9-9160-948C0B0F796F@stufft.io> Message-ID: <5DE48AF4-0047-4739-BECF-EB15E5E6D565@stufft.io> > On Oct 27, 2014, at 4:45 PM, Daniel Holth wrote: > > I liked it because I agree with the TOML author that the YAML spec > gives rage; YAML seems to be defined as a bunch of things that the end > user is supposed to think are intuitive, but try understanding and > correctly parsing the full set of what is allowed... TOML on the other > hand is short. > > On Mon, Oct 27, 2014 at 4:23 PM, Paul Moore wrote: >> On 27 October 2014 19:23, Donald Stufft wrote: >>> Ugh, I hate TOML. I?m -1 on any of the standards using it, but I also >>> think the standards should be around data exchange and should just use >>> JSON and leave front end stuff like that up to the implementations. >> >> I had a quick glance at TOML, and I can't say I was particularly >> enamoured by it. I don't see that it has any particularly huge >> benefits over "plain" ini files (if your needs are simple) or YAML >> (ignoring the over-complicated stuff that nobody actually needs). >> >> +1 on JSON for "internal" format, and tools deciding for themselves on >> the best user-facing format. >> >> I'm also not sure I see the value of mapping directly to a dict. >> Surely internal formats should be isolated from the user interface, >> not exposed directly? >> Paul The YAML spec isn?t for end users any more than the various HTTP RFCs are for end users. The spec is for people implementing a yam parser/encoder and when implementing a spec the less ambiguity and the more verbose the spec is, the better. It?s not a very good argument though, because JSON is the better format for data exchange and that?s all the standards should be focused on. If someone wants to make a tool that uses TOML and emits standard metadata (when that becomes a thing outside of Wheels) more power to them. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From holger at merlinux.eu Tue Oct 28 10:59:08 2014 From: holger at merlinux.eu (holger krekel) Date: Tue, 28 Oct 2014 09:59:08 +0000 Subject: [Distutils] toml In-Reply-To: References: <33684B52-DC82-4DD9-9160-948C0B0F796F@stufft.io> Message-ID: <20141028095908.GA22258@merlinux.eu> On Mon, Oct 27, 2014 at 16:45 -0400, Daniel Holth wrote: > I liked it because I agree with the TOML author that the YAML spec > gives rage; YAML seems to be defined as a bunch of things that the end > user is supposed to think are intuitive, but try understanding and > correctly parsing the full set of what is allowed... TOML on the other > hand is short. Don't know but TOML could work well as a user facing config syntax. Seems to be reasonably well defined and more expressive than plain ini files. Would be curious if it could be used for tox for example. cheers, holger > On Mon, Oct 27, 2014 at 4:23 PM, Paul Moore wrote: > > On 27 October 2014 19:23, Donald Stufft wrote: > >> Ugh, I hate TOML. I?m -1 on any of the standards using it, but I also > >> think the standards should be around data exchange and should just use > >> JSON and leave front end stuff like that up to the implementations. > > > > I had a quick glance at TOML, and I can't say I was particularly > > enamoured by it. I don't see that it has any particularly huge > > benefits over "plain" ini files (if your needs are simple) or YAML > > (ignoring the over-complicated stuff that nobody actually needs). > > > > +1 on JSON for "internal" format, and tools deciding for themselves on > > the best user-facing format. > > > > I'm also not sure I see the value of mapping directly to a dict. > > Surely internal formats should be isolated from the user interface, > > not exposed directly? > > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From solipsis at pitrou.net Tue Oct 28 13:57:47 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 28 Oct 2014 13:57:47 +0100 Subject: [Distutils] Making a wheel platform-specific? Message-ID: <20141028135747.50c1b5ef@fsol> Hello, I am trying to package a project which has the following features: - pure Python code (2- and 3-compatible) - no C extensions (therefore no CPython ABI issues) - but there's a .so file (or DLL) that's loaded at runtime using ctypes (FYI: https://github.com/numba/llvmlite) I would like to tell bdist_wheel to therefore tag the package as a Python-independent, platform-dependent binary package. There doesn't seem to be a way of doing so, or is there? Regards Antoine. From p.f.moore at gmail.com Tue Oct 28 14:15:09 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 28 Oct 2014 13:15:09 +0000 Subject: [Distutils] Making a wheel platform-specific? In-Reply-To: <20141028135747.50c1b5ef@fsol> References: <20141028135747.50c1b5ef@fsol> Message-ID: On 28 October 2014 12:57, Antoine Pitrou wrote: > I would like to tell bdist_wheel to therefore tag the package as a > Python-independent, platform-dependent binary package. There doesn't > seem to be a way of doing so, or is there? I don't believe there is at the moment. But the tags are *only* recorded in the filename, so it's perfectly OK to simply rename the wheel file manually after it's created. It would be nice to file a feature request for this on the wheel tracker (https://bitbucket.org/pypa/wheel) as well, if there's nothing there already, so we don't forget. Paul From solipsis at pitrou.net Tue Oct 28 14:20:21 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 28 Oct 2014 14:20:21 +0100 Subject: [Distutils] Making a wheel platform-specific? In-Reply-To: References: <20141028135747.50c1b5ef@fsol> Message-ID: <20141028142021.758af8ff@fsol> On Tue, 28 Oct 2014 13:15:09 +0000 Paul Moore wrote: > On 28 October 2014 12:57, Antoine Pitrou wrote: > > I would like to tell bdist_wheel to therefore tag the package as a > > Python-independent, platform-dependent binary package. There doesn't > > seem to be a way of doing so, or is there? > > I don't believe there is at the moment. But the tags are *only* > recorded in the filename, so it's perfectly OK to simply rename the > wheel file manually after it's created. > > It would be nice to file a feature request for this on the wheel > tracker (https://bitbucket.org/pypa/wheel) as well, if there's nothing > there already, so we don't forget. Ok, will do. I was assuming the compatibility tags would be recorded somewhere in the metadata. If I rename the file manually, is there an easy (CLI-based) way of uploading it to PyPI? Also, there's a remaining issue: how do I tell wheel that the package needs platlib rather than purelib? (may that question also apply to other bdist* flavours?) Regards Antoine. From solipsis at pitrou.net Tue Oct 28 14:23:57 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 28 Oct 2014 14:23:57 +0100 Subject: [Distutils] Making a wheel platform-specific? In-Reply-To: References: <20141028135747.50c1b5ef@fsol> Message-ID: <20141028142357.5546be12@fsol> On Tue, 28 Oct 2014 13:15:09 +0000 Paul Moore wrote: > On 28 October 2014 12:57, Antoine Pitrou wrote: > > I would like to tell bdist_wheel to therefore tag the package as a > > Python-independent, platform-dependent binary package. There doesn't > > seem to be a way of doing so, or is there? > > I don't believe there is at the moment. But the tags are *only* > recorded in the filename, so it's perfectly OK to simply rename the > wheel file manually after it's created. > > It would be nice to file a feature request for this on the wheel > tracker (https://bitbucket.org/pypa/wheel) as well, if there's nothing > there already, so we don't forget. Ok, the issue already existed: https://bitbucket.org/pypa/wheel/issue/100/binary-wheels-for-different-python Regards Antoine. From olivier.grisel at ensta.org Tue Oct 28 14:39:05 2014 From: olivier.grisel at ensta.org (Olivier Grisel) Date: Tue, 28 Oct 2014 14:39:05 +0100 Subject: [Distutils] Making a wheel platform-specific? In-Reply-To: <20141028142021.758af8ff@fsol> References: <20141028135747.50c1b5ef@fsol> <20141028142021.758af8ff@fsol> Message-ID: 2014-10-28 14:20 GMT+01:00 Antoine Pitrou : > > If I rename the file manually, is there an easy (CLI-based) way of > uploading it to PyPI? I had the same issue for a tool that collects wheels generated by various CI platforms to prepare a binary release. https://github.com/ogrisel/wheelhouse-uploader Here is the hack I did (a distutils command extension): https://github.com/ogrisel/wheelhouse-uploader/blob/master/wheelhouse_uploader/cmd.py#L54 -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel From p.f.moore at gmail.com Tue Oct 28 14:39:32 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 28 Oct 2014 13:39:32 +0000 Subject: [Distutils] Making a wheel platform-specific? In-Reply-To: <20141028142021.758af8ff@fsol> References: <20141028135747.50c1b5ef@fsol> <20141028142021.758af8ff@fsol> Message-ID: On 28 October 2014 13:20, Antoine Pitrou wrote: > If I rename the file manually, is there an easy (CLI-based) way of > uploading it to PyPI? I think twine can do that for you (and is generally recommended these days over setup.py upload, as it uses https). > Also, there's a remaining issue: how do I tell wheel that the package > needs platlib rather than purelib? > (may that question also apply to other bdist* flavours?) You're right that's probably a more general question. For wheels, there's a "Root-Is-Purelib" entry in the WHEEL file in the metadata. That one would need manual hacking, unfortunately. Paul From hafner87 at gmail.com Tue Oct 28 11:13:40 2014 From: hafner87 at gmail.com (Matthias Hafner) Date: Tue, 28 Oct 2014 11:13:40 +0100 Subject: [Distutils] PEP425 - Wheel Binary Package Compatibility Message-ID: Hi there, Following up on https://bitbucket.org/pypa/wheel/issue/124/glibc-incompatibility. How should we deal with incompatibility of dynamically linked libraries? Doing "pip wheel numpy" on a Linux 64bit machine results in , which is linked dynamically against the GLIBC version installed on the build machine. So should the wheel be shipped with GLIBC, or the GLIBC version be specified in the wheel name (means to build a new wheel for each GLIBC version?). Also, maybe this is not the only library it is dynamically linked against? Why does that work on MacOS, btw? Are all library versions fixed there for one version of OSX? Thanks for putting some light into this issue. Best regards, Matthias -------------- next part -------------- An HTML attachment was scrubbed... URL: From p at 2014.dobrogost.net Tue Oct 28 11:43:30 2014 From: p at 2014.dobrogost.net (Piotr Dobrogost) Date: Tue, 28 Oct 2014 11:43:30 +0100 Subject: [Distutils] toml In-Reply-To: <20141028095908.GA22258@merlinux.eu> References: <33684B52-DC82-4DD9-9160-948C0B0F796F@stufft.io> <20141028095908.GA22258@merlinux.eu> Message-ID: There's also AXON ("It tries to combine the best of JSON, XML and YAML.") ? http://intellimath.bitbucket.org/axon/ From solipsis at pitrou.net Tue Oct 28 14:43:02 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 28 Oct 2014 14:43:02 +0100 Subject: [Distutils] Making a wheel platform-specific? In-Reply-To: References: <20141028135747.50c1b5ef@fsol> <20141028142021.758af8ff@fsol> Message-ID: <20141028144302.085bd7af@fsol> On Tue, 28 Oct 2014 13:39:32 +0000 Paul Moore wrote: > On 28 October 2014 13:20, Antoine Pitrou wrote: > > If I rename the file manually, is there an easy (CLI-based) way of > > uploading it to PyPI? > > I think twine can do that for you (and is generally recommended these > days over setup.py upload, as it uses https). setup.py upload also uses https these days, AFAIK. But I'll take a look at twine (*sigh*... another tool). Regards Antoine. From donald at stufft.io Tue Oct 28 15:04:17 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 28 Oct 2014 10:04:17 -0400 Subject: [Distutils] Making a wheel platform-specific? In-Reply-To: <20141028144302.085bd7af@fsol> References: <20141028135747.50c1b5ef@fsol> <20141028142021.758af8ff@fsol> <20141028144302.085bd7af@fsol> Message-ID: <3C41CF03-40EA-4665-AC81-473D724BC27E@stufft.io> > On Oct 28, 2014, at 9:43 AM, Antoine Pitrou wrote: > >> I think twine can do that for you (and is generally recommended these >> days over setup.py upload, as it uses https). > > setup.py upload also uses https these days, AFAIK. Paul forgot an important word there, *verified* HTTPS. It is true that recently Python switched the default from http:// to https:// however in any released version of Python it still does not *verify* that when uploading. I believe that PEP 476 will change that but it?s not in any released Python to my knowledge and even then it?s hard to tell people ?Well setup.py upload is safe, in these particular scenarios? when it?s hard to declare exactly when those scenarios are (Python versions, but also some older versions thanks to downstream back porting patches etc). It doesn?t help either that the design of distutils doesn?t allow uploading an existing file so if you, for instance, want to upload a Python 2.6 specific Wheel then that won?t have PEP 476 and then something like twine is the only way forward. All in all it?s easier to just say ?don?t use that, use this instead? than to explain when and why it?s ok to use that. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From solipsis at pitrou.net Tue Oct 28 15:04:19 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 28 Oct 2014 15:04:19 +0100 Subject: [Distutils] twine: Binary wheel for an unsupported platform Message-ID: <20141028150419.782aa0e7@fsol> Hello, I guess I am doing something wrong, but what? (I took "linux_x86_64" after PEP 427 and 425) $ twine upload dist/llvmlite-0.1-py2.py3-none-linux_x86_64.whl Uploading distributions to https://pypi.python.org/pypi Uploading llvmlite-0.1-py2.py3-none-linux_x86_64.whl Traceback (most recent call last): File "/home/antoine/.local/lib/python3.4/site-packages/twine/commands/upload.py", line 228, in main upload(**vars(args)) File "/home/antoine/.local/lib/python3.4/site-packages/twine/commands/upload.py", line 183, in upload resp.raise_for_status() File "/home/antoine/.local/lib/python3.4/site-packages/requests/models.py", line 825, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 400 Client Error: Binary wheel for an unsupported platform During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/antoine/.local/bin/twine-upload", line 11, in sys.exit(main()) File "/home/antoine/.local/lib/python3.4/site-packages/twine/commands/upload.py", line 230, in main sys.exit("{0}: {1}".format(exc.__class__.__name__, exc.message)) AttributeError: 'HTTPError' object has no attribute 'message' (also note the Python 3 buglet in main() above) Regards Antoine. From p.f.moore at gmail.com Tue Oct 28 15:04:30 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 28 Oct 2014 14:04:30 +0000 Subject: [Distutils] PEP425 - Wheel Binary Package Compatibility In-Reply-To: References: Message-ID: On 28 October 2014 10:13, Matthias Hafner wrote: (in reverse order, easiest to hardest :-)) > Why does that work on MacOS, btw? Are all library versions fixed there for > one version of OSX? I believe (I'm not 100% sure as I'm a Windows user not an OSX user) that PyPI only supports binaries compatible with the official python.org binaries of Python. So library version decisions are moot. This means that homebrew or "the other ones" (sorry :-)) users can't avail themselves of PyPI-hosted binaries yet, until the wider library compatibility issues are resolved. > So should the wheel be shipped with GLIBC, or the GLIBC version be specified > in the wheel name (means to build a new wheel for each GLIBC version?). > Also, maybe this is not the only library it is dynamically linked against? Whatever solution is devised will need to be encapsulated in a set of compatibility tags (see PEP 425) that capture what it means to be "compatible". The wheels can then be uploaded with the correct tags, and pip and other tools will automatically select the right one (or report an error if no suitable binary exists and source is not available). > How should we deal with incompatibility of dynamically linked libraries? > Doing "pip wheel numpy" on a Linux 64bit machine results in , which is > linked dynamically against the GLIBC version installed on the build machine. This is the big question. On Windows, we only really have to deal with 32-bit or 64-bit binaries, using the C runtime that the python.org binaries use. So the architecture is sufficient and the ABI flag is unused. On OSX, as I say, PyPI has in effect artificially set up a similar situation, by not considering anything other than python.org build compatible binaries. On Linux, nobody has really looked at the problem (beyond acknowledging that it exists). We need a group of Linux experts with an understanding of how all the various aspects of distribution, kernel, libc, etc, work together to form a "compatible environment" for binaries. As Linux has a strong culture of only building from source or using distro packages, there doesn't seem to me (as an outsider) to be much momentum around solving this. Sorry I can't offer any more help or clarification than this. Hopefully someone from the Linux side of things can offer some views on how this can be moved forwards. Paul From solipsis at pitrou.net Tue Oct 28 15:06:20 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 28 Oct 2014 15:06:20 +0100 Subject: [Distutils] Making a wheel platform-specific? In-Reply-To: <3C41CF03-40EA-4665-AC81-473D724BC27E@stufft.io> References: <20141028135747.50c1b5ef@fsol> <20141028142021.758af8ff@fsol> <20141028144302.085bd7af@fsol> <3C41CF03-40EA-4665-AC81-473D724BC27E@stufft.io> Message-ID: <20141028150620.53e1dcd4@fsol> On Tue, 28 Oct 2014 10:04:17 -0400 Donald Stufft wrote: > > > On Oct 28, 2014, at 9:43 AM, Antoine Pitrou wrote: > > > >> I think twine can do that for you (and is generally recommended these > >> days over setup.py upload, as it uses https). > > > > setup.py upload also uses https these days, AFAIK. > > Paul forgot an important word there, *verified* HTTPS. Yes, you're right. Thanks for doing this. Regards Antoine. From p.f.moore at gmail.com Tue Oct 28 15:07:03 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 28 Oct 2014 14:07:03 +0000 Subject: [Distutils] twine: Binary wheel for an unsupported platform In-Reply-To: <20141028150419.782aa0e7@fsol> References: <20141028150419.782aa0e7@fsol> Message-ID: On 28 October 2014 14:04, Antoine Pitrou wrote: > I guess I am doing something wrong, but what? > (I took "linux_x86_64" after PEP 427 and 425) Oops, I mentioned this in another thread but forgot it when responding to you. Sorry. PyPI does not currently allow uploading binary wheels for Linux, because the "compatibility tags" issues are unresolved, and the decision was taken to block binary uploads rather than cause user confusion when pip started installing unusable binaries. Paul From p.f.moore at gmail.com Tue Oct 28 15:07:47 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 28 Oct 2014 14:07:47 +0000 Subject: [Distutils] Making a wheel platform-specific? In-Reply-To: <20141028150620.53e1dcd4@fsol> References: <20141028135747.50c1b5ef@fsol> <20141028142021.758af8ff@fsol> <20141028144302.085bd7af@fsol> <3C41CF03-40EA-4665-AC81-473D724BC27E@stufft.io> <20141028150620.53e1dcd4@fsol> Message-ID: >> Paul forgot an important word there, *verified* HTTPS. Paul always forgets the important details when discussing security :-) Sorry. Paul. From solipsis at pitrou.net Tue Oct 28 15:22:00 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 28 Oct 2014 15:22:00 +0100 Subject: [Distutils] "not a supported wheel on this platform" Message-ID: <20141028152200.168e7930@fsol> Hello, Ok, so until I find a better solution I've tried hosting a wheel on a personal server, but then I get the following error when installing it: $ pyvenv-3.4 t $ source t/bin/activate (t) $ pip install https://ssl.pitrou.net/llvmlite-0.1-py2.py3-none-linux_x86_64.whl llvmlite-0.1-py2.py3-none-linux_x86_64.whl is not a supported wheel on this platform. Storing debug log for failure in /home/antoine/.pip/pip.log Yet: (t) $ python Python 3.4.1 (3.4:d1bf37def4fd, May 19 2014, 19:52:53) [GCC 4.8.1] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import sysconfig >>> sysconfig.get_platform() 'linux-x86_64' The log file doesn't tell anything really interesting: ------------------------------------------------------------ /home/antoine/t/t/bin/pip run on Tue Oct 28 15:18:06 2014 llvmlite-0.1-py2.py3-none-linux_x86_64.whl is not a supported wheel on this platform. Exception information: Traceback (most recent call last): File "/home/antoine/t/t/lib/python3.4/site-packages/pip/basecommand.py", line 122, in main status = self.run(options, args) File "/home/antoine/t/t/lib/python3.4/site-packages/pip/commands/install.py", line 257, in run InstallRequirement.from_line(name, None)) File "/home/antoine/t/t/lib/python3.4/site-packages/pip/req.py", line 167, in from_line raise UnsupportedWheel("%s is not a supported wheel on this platform." % wheel.filename) pip.exceptions.UnsupportedWheel: llvmlite-0.1-py2.py3-none-linux_x86_64.whl is not a supported wheel on this platform. Regards Antoine. From chris.jerdonek at gmail.com Tue Oct 28 15:43:41 2014 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Tue, 28 Oct 2014 07:43:41 -0700 Subject: [Distutils] toml In-Reply-To: <20141028095908.GA22258@merlinux.eu> References: <33684B52-DC82-4DD9-9160-948C0B0F796F@stufft.io> <20141028095908.GA22258@merlinux.eu> Message-ID: On Tue, Oct 28, 2014 at 2:59 AM, holger krekel wrote: > On Mon, Oct 27, 2014 at 16:45 -0400, Daniel Holth wrote: >> I liked it because I agree with the TOML author that the YAML spec >> gives rage; YAML seems to be defined as a bunch of things that the end >> user is supposed to think are intuitive, but try understanding and >> correctly parsing the full set of what is allowed... TOML on the other >> hand is short. > > Don't know but TOML could work well as a user facing config syntax. > Seems to be reasonably well defined and more expressive than plain ini > files. Would be curious if it could be used for tox for example. Daniel, Holger, et al., what is wrong with ini out of curiosity? Thanks, --Chris From p.f.moore at gmail.com Tue Oct 28 15:44:22 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 28 Oct 2014 14:44:22 +0000 Subject: [Distutils] "not a supported wheel on this platform" In-Reply-To: <20141028152200.168e7930@fsol> References: <20141028152200.168e7930@fsol> Message-ID: Can you see what the following says (results here are from my Windows Python 3.4 system)? >>> from wheel.pep425tags import get_supported >>> ['-'.join(t) for t in get_supported()] ['cp34-none-win_amd64', 'cp34-none-any', 'cp3-none-any', 'cp33-none-any', 'cp32-none-any', 'cp31-none-any', 'cp30-none-any', 'py34-none-any', 'py3-none-any', 'py33-none-any', 'py32-none-any', 'py31-none-any', 'py30-none-any'] I suspect you might be missing the combination "py2" with an architecture other than "any". Paul On 28 October 2014 14:22, Antoine Pitrou wrote: > > Hello, > > Ok, so until I find a better solution I've tried hosting a wheel on a > personal server, but then I get the following error when installing it: > > $ pyvenv-3.4 t > $ source t/bin/activate > (t) $ pip install https://ssl.pitrou.net/llvmlite-0.1-py2.py3-none-linux_x86_64.whl > llvmlite-0.1-py2.py3-none-linux_x86_64.whl is not a supported wheel on this platform. > Storing debug log for failure in /home/antoine/.pip/pip.log > > Yet: > > (t) $ python > Python 3.4.1 (3.4:d1bf37def4fd, May 19 2014, 19:52:53) > [GCC 4.8.1] on linux > Type "help", "copyright", "credits" or "license" for more information. >>>> import sysconfig >>>> sysconfig.get_platform() > 'linux-x86_64' > > > The log file doesn't tell anything really interesting: > > ------------------------------------------------------------ > /home/antoine/t/t/bin/pip run on Tue Oct 28 15:18:06 2014 > llvmlite-0.1-py2.py3-none-linux_x86_64.whl is not a supported wheel on this platform. > Exception information: > Traceback (most recent call last): > File "/home/antoine/t/t/lib/python3.4/site-packages/pip/basecommand.py", line 122, in main > status = self.run(options, args) > File "/home/antoine/t/t/lib/python3.4/site-packages/pip/commands/install.py", line 257, in run > InstallRequirement.from_line(name, None)) > File "/home/antoine/t/t/lib/python3.4/site-packages/pip/req.py", line 167, in from_line > raise UnsupportedWheel("%s is not a supported wheel on this platform." % wheel.filename) > pip.exceptions.UnsupportedWheel: llvmlite-0.1-py2.py3-none-linux_x86_64.whl is not a supported wheel on this platform. > > > Regards > > Antoine. > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig From solipsis at pitrou.net Tue Oct 28 15:51:55 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 28 Oct 2014 15:51:55 +0100 Subject: [Distutils] "not a supported wheel on this platform" In-Reply-To: References: <20141028152200.168e7930@fsol> Message-ID: <20141028155155.5579bd9f@fsol> On Tue, 28 Oct 2014 14:44:22 +0000 Paul Moore wrote: > Can you see what the following says (results here are from my Windows > Python 3.4 system)? > > >>> from wheel.pep425tags import get_supported > >>> ['-'.join(t) for t in get_supported()] > ['cp34-none-win_amd64', 'cp34-none-any', 'cp3-none-any', > 'cp33-none-any', 'cp32-none-any', 'cp31-none-any', 'cp30-none-any', > 'py34-none-any', 'py3-none-any', 'py33-none-any', 'py32-none-any', > 'py31-none-any', 'py30-none-any'] > > I suspect you might be missing the combination "py2" with an > architecture other than "any". Does matching really happen that way? Here it is anyway. There's no "py2" at all: >>> ['-'.join(t) for t in get_supported()] ['cp34-cp34m-linux_x86_64', 'cp34-abi3-linux_x86_64', 'cp34-none-linux_x86_64', 'cp34-none-any', 'cp3-none-any', 'cp33-none-any', 'cp32-none-any', 'cp31-none-any', 'cp30-none-any', 'py34-none-any', 'py3-none-any', 'py33-none-any', 'py32-none-any', 'py31-none-any', 'py30-none-any'] Regards Antoine. From chris.jerdonek at gmail.com Tue Oct 28 15:58:31 2014 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Tue, 28 Oct 2014 07:58:31 -0700 Subject: [Distutils] twine: Binary wheel for an unsupported platform In-Reply-To: <20141028150419.782aa0e7@fsol> References: <20141028150419.782aa0e7@fsol> Message-ID: On Tue, Oct 28, 2014 at 7:04 AM, Antoine Pitrou wrote: > > Hello, > > I guess I am doing something wrong, but what? > (I took "linux_x86_64" after PEP 427 and 425) > > > $ twine upload dist/llvmlite-0.1-py2.py3-none-linux_x86_64.whl > Uploading distributions to https://pypi.python.org/pypi > Uploading llvmlite-0.1-py2.py3-none-linux_x86_64.whl > Traceback (most recent call last): > File "/home/antoine/.local/lib/python3.4/site-packages/twine/commands/upload.py", line 228, in main > upload(**vars(args)) > File "/home/antoine/.local/lib/python3.4/site-packages/twine/commands/upload.py", line 183, in upload > resp.raise_for_status() > File "/home/antoine/.local/lib/python3.4/site-packages/requests/models.py", line 825, in raise_for_status > raise HTTPError(http_error_msg, response=self) > requests.exceptions.HTTPError: 400 Client Error: Binary wheel for an unsupported platform > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/home/antoine/.local/bin/twine-upload", line 11, in > sys.exit(main()) > File "/home/antoine/.local/lib/python3.4/site-packages/twine/commands/upload.py", line 230, in main > sys.exit("{0}: {1}".format(exc.__class__.__name__, exc.message)) > AttributeError: 'HTTPError' object has no attribute 'message' > > > (also note the Python 3 buglet in main() above) FYI, it looks like this has been fixed already: https://github.com/pypa/twine/commit/162ab0e9d0d93a2357a6440bdf7af1835247e025 --Chris From cournape at gmail.com Tue Oct 28 16:01:42 2014 From: cournape at gmail.com (David Cournapeau) Date: Tue, 28 Oct 2014 15:01:42 +0000 Subject: [Distutils] PEP425 - Wheel Binary Package Compatibility In-Reply-To: References: Message-ID: On Tue, Oct 28, 2014 at 10:13 AM, Matthias Hafner wrote: > Hi there, > > Following up on > https://bitbucket.org/pypa/wheel/issue/124/glibc-incompatibility. > > How should we deal with incompatibility of dynamically linked libraries? > Doing "pip wheel numpy" on a Linux 64bit machine results in , which is > linked dynamically against the GLIBC version installed on the build > machine. > Wheel by itself simply cannot deal with this, you need something more. Practically speaking, there is no such a thing as ABI on Linux: even if you somehow managed to deal with glibc, you would then need to deal with fortran ABI, then with C++ ABI, etc... Dealing with this at the python level is simply intractable. That needs to be handled at the distribution level, or within a "sumo distribution" like Anaconda from Continuum or Canopy from Enthought (disclaimer, I work for Enthought), which makes the problem more tractable by picking up an ABI and stick with it for the full set of packages. In theory, you can try to emulate this by building on the oldest version of each compiled library, and hope that your target platforms are ABI compatible with this, but that's hard on Linux. David > So should the wheel be shipped with GLIBC, or the GLIBC version be > specified in the wheel name (means to build a new wheel for each GLIBC > version?). Also, maybe this is not the only library it is dynamically > linked against? > > Why does that work on MacOS, btw? Are all library versions fixed there for > one version of OSX? > Yes. Same for windows. The native toolchains there are fairly easy to configure for targetting a specific minimum version. David > Thanks for putting some light into this issue. > > Best regards, > Matthias > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Oct 28 16:03:23 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 28 Oct 2014 15:03:23 +0000 Subject: [Distutils] "not a supported wheel on this platform" In-Reply-To: <20141028155155.5579bd9f@fsol> References: <20141028152200.168e7930@fsol> <20141028155155.5579bd9f@fsol> Message-ID: On 28 October 2014 14:51, Antoine Pitrou wrote: > Does matching really happen that way? Unfortunately yes :-( get_supported() introspects the system you're installing on and builds a list of tag combinations it's willing to accept. In doing so it misses some that I'd consider obscure but reasonable (notably here, independent of Python implementation (py rather than cp) but architecture specific). I consider this a flawed design, but it's always been like this. You'll need to raise a bug report for this. Actually two, as the pep425tags code is in both pip and wheel independently (the one you actually *want* is the pip one, though). I know - this should really be fixed. I'd like to see pep425tags in a separate project, referenced (or vendored if necessary) from both pip and wheel. And I'd like a matching algorithm for checking tags rather than generating a list of valid values. Oh, and a pony :-) > Here it is anyway. There's no "py2" at all: Sorry, from any given Python you'll get either py2 or py3 depending on the major version of your Python. Paul From holger at merlinux.eu Tue Oct 28 16:09:49 2014 From: holger at merlinux.eu (holger krekel) Date: Tue, 28 Oct 2014 15:09:49 +0000 Subject: [Distutils] toml In-Reply-To: References: <33684B52-DC82-4DD9-9160-948C0B0F796F@stufft.io> <20141028095908.GA22258@merlinux.eu> Message-ID: <20141028150949.GD22258@merlinux.eu> On Tue, Oct 28, 2014 at 07:43 -0700, Chris Jerdonek wrote: > On Tue, Oct 28, 2014 at 2:59 AM, holger krekel wrote: > > On Mon, Oct 27, 2014 at 16:45 -0400, Daniel Holth wrote: > >> I liked it because I agree with the TOML author that the YAML spec > >> gives rage; YAML seems to be defined as a bunch of things that the end > >> user is supposed to think are intuitive, but try understanding and > >> correctly parsing the full set of what is allowed... TOML on the other > >> hand is short. > > > > Don't know but TOML could work well as a user facing config syntax. > > Seems to be reasonably well defined and more expressive than plain ini > > files. Would be curious if it could be used for tox for example. > > Daniel, Holger, et al., what is wrong with ini out of curiosity? E.g. There are no types with normal ini files, so integers, strings, lists etc. all have to be done manually, i.e. without support from the parser. best, holger > Thanks, > --Chris > From solipsis at pitrou.net Tue Oct 28 16:15:16 2014 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 28 Oct 2014 16:15:16 +0100 Subject: [Distutils] "not a supported wheel on this platform" In-Reply-To: References: <20141028152200.168e7930@fsol> <20141028155155.5579bd9f@fsol> Message-ID: <20141028161516.77b8c6fb@fsol> On Tue, 28 Oct 2014 15:03:23 +0000 Paul Moore wrote: > On 28 October 2014 14:51, Antoine Pitrou wrote: > > Does matching really happen that way? > > Unfortunately yes :-( > > get_supported() introspects the system you're installing on and builds > a list of tag combinations it's willing to accept. In doing so it > misses some that I'd consider obscure but reasonable (notably here, > independent of Python implementation (py rather than cp) but > architecture specific). I consider this a flawed design, but it's > always been like this. > > You'll need to raise a bug report for this. Actually two, as the > pep425tags code is in both pip and wheel independently (the one you > actually *want* is the pip one, though). Apparently already reported to pip :-) https://github.com/pypa/pip/issues/1870 Reported to wheel at https://bitbucket.org/pypa/wheel/issue/125/get_supported-should-return-more Thanks for the explanations. Regards Antoine. From olivier.grisel at ensta.org Tue Oct 28 16:20:38 2014 From: olivier.grisel at ensta.org (Olivier Grisel) Date: Tue, 28 Oct 2014 16:20:38 +0100 Subject: [Distutils] PEP425 - Wheel Binary Package Compatibility In-Reply-To: References: Message-ID: 2014-10-28 15:04 GMT+01:00 Paul Moore : > On 28 October 2014 10:13, Matthias Hafner wrote: > > (in reverse order, easiest to hardest :-)) > >> Why does that work on MacOS, btw? Are all library versions fixed there for >> one version of OSX? > > I believe (I'm not 100% sure as I'm a Windows user not an OSX user) > that PyPI only supports binaries compatible with the official > python.org binaries of Python. So library version decisions are moot. > This means that homebrew or "the other ones" (sorry :-)) users can't > avail themselves of PyPI-hosted binaries yet, until the wider library > compatibility issues are resolved. Homebrew-installed Python and the system Python that comes by default on OSX are binary compatible with wheels generated with Python from the official OSX installer from python.org. In current versions of pip you "just" have to rename the wheel filenames to get them picked up from PyPI on those platforms. This explains the strange names of the OSX wheels for numpy for instance: numpy-1.9.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl numpy-1.9.0-cp33-cp33m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl numpy-1.9.0-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl https://pypi.python.org/pypi/numpy macosx_10_9_x86_64 is the platform tag used by Homebrew-installed Python while macosx_10_6_intel is the platform tag of the python.org Python. Off-course if your python extension link to non-system libraries, you have to embed them in the wheel, for instance using: https://pypi.python.org/pypi/delocate -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel From olivier.grisel at ensta.org Tue Oct 28 16:22:16 2014 From: olivier.grisel at ensta.org (Olivier Grisel) Date: Tue, 28 Oct 2014 16:22:16 +0100 Subject: [Distutils] PEP425 - Wheel Binary Package Compatibility In-Reply-To: References: Message-ID: Here is a test matrix that checks binary compatibility of OSX wheels for many projects of the SciPy stack: https://github.com/MacPython/scipy-stack-osx-testing -- Olivier From olivier.grisel at ensta.org Tue Oct 28 16:24:06 2014 From: olivier.grisel at ensta.org (Olivier Grisel) Date: Tue, 28 Oct 2014 16:24:06 +0100 Subject: [Distutils] PEP425 - Wheel Binary Package Compatibility In-Reply-To: References: Message-ID: 2014-10-28 16:22 GMT+01:00 Olivier Grisel : > Here is a test matrix that checks binary compatibility of OSX wheels > for many projects of the SciPy stack: > > https://github.com/MacPython/scipy-stack-osx-testing And here are the results: https://travis-ci.org/MacPython/scipy-stack-osx-testing -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel From p.f.moore at gmail.com Tue Oct 28 16:26:03 2014 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 28 Oct 2014 15:26:03 +0000 Subject: [Distutils] "not a supported wheel on this platform" In-Reply-To: <20141028161516.77b8c6fb@fsol> References: <20141028152200.168e7930@fsol> <20141028155155.5579bd9f@fsol> <20141028161516.77b8c6fb@fsol> Message-ID: On 28 October 2014 15:15, Antoine Pitrou wrote: > Apparently already reported to pip :-) > https://github.com/pypa/pip/issues/1870 I *knew* I'd seen it somewhere. Thanks for locating it. > Reported to wheel at > https://bitbucket.org/pypa/wheel/issue/125/get_supported-should-return-more And thanks again. Paul From mal at egenix.com Tue Oct 28 16:47:25 2014 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 28 Oct 2014 16:47:25 +0100 Subject: [Distutils] PEP425 - Wheel Binary Package Compatibility In-Reply-To: References: Message-ID: <544FBA8D.2040804@egenix.com> On 28.10.2014 11:13, Matthias Hafner wrote: > Hi there, > > Following up on > https://bitbucket.org/pypa/wheel/issue/124/glibc-incompatibility. > > How should we deal with incompatibility of dynamically linked libraries? > Doing "pip wheel numpy" on a Linux 64bit machine results in , which is > linked dynamically against the GLIBC version installed on the build > machine. > > So should the wheel be shipped with GLIBC, or the GLIBC version be > specified in the wheel name (means to build a new wheel for each GLIBC > version?). Also, maybe this is not the only library it is dynamically > linked against? > > Why does that work on MacOS, btw? Are all library versions fixed there for > one version of OSX? > > Thanks for putting some light into this issue. Since OSes typically ship with older libc versions, your best bet is to build the package on a OS release that's a few years older than the latest version, e.g. for Ubuntu you'd use 12.04 or even 10.04 instead of 14.04. On Linux, you can check the min required GLIBC version by looking at the nm output of the binaries. They will have a "@@GLIBC_x.x.x" modifier attached to the symbols loaded from the GLIBC. The platform module has a helper function which does this for you: https://docs.python.org/2.7/library/platform.html#platform.libc_ver That way you can avoid many incompatibilities with libc versions. This works on Mac OS X and other Unix-based systems as well. You may run into library version info problems, though: http://stackoverflow.com/questions/137773/what-does-the-no-version-information-available-error-from-linux-dynamic-linker/156387#156387 A way around this is to build on systems that don't yet include this version information. The alternative is static linking, but this is often not possible or desired. On Windows you have to use the libc versions that were used to compile Python itself, so things are easier. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 28 2014) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2014-10-24: Released eGenix pyOpenSSL 0.13.5 ... http://egenix.com/go63 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ethan at stoneleaf.us Tue Oct 28 19:20:46 2014 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 28 Oct 2014 11:20:46 -0700 Subject: [Distutils] a package with is a module In-Reply-To: References: <544E4D01.3030900@stoneleaf.us> <544E5FBF.9040406@stoneleaf.us> Message-ID: <544FDE7E.1060901@stoneleaf.us> On 10/27/2014 08:26 AM, Paul Moore wrote: > On 27 October 2014 15:07, Ethan Furman wrote: >> Paul Moore also declaimed: >>> Alternatively, you could distribute all 3 files [...] >> >> The problem I have with this method is that the last time I tried it >> setup.py attempted to pre-compile all the files, resulting in syntax errors, >> which makes for a lousy user experience. > > Yeah, it's harmless but ugly. I've seen that issue myself. For the record, I thought setting PYTHONDONTWRITEBYTECODE from inside the setup.py script: os.environ['PYTHONDONTWRITEBYTECODE'] = True might do the trick, but - it isn't available on 2.5 - it doesn't work on 2.6 (didn't test on 2.7 nor 3.x) Oh well. Guess I'll include a note that says, "Ignore any syntax errors during 'setup.py install', they're harmless." :( -- ~Ethan~ From donald at stufft.io Tue Oct 28 19:22:27 2014 From: donald at stufft.io (Donald Stufft) Date: Tue, 28 Oct 2014 14:22:27 -0400 Subject: [Distutils] a package with is a module In-Reply-To: <544FDE7E.1060901@stoneleaf.us> References: <544E4D01.3030900@stoneleaf.us> <544E5FBF.9040406@stoneleaf.us> <544FDE7E.1060901@stoneleaf.us> Message-ID: <64226516-DE65-4168-8E12-ADE8864E4A1A@stufft.io> > On Oct 28, 2014, at 2:20 PM, Ethan Furman wrote: > > On 10/27/2014 08:26 AM, Paul Moore wrote: >> On 27 October 2014 15:07, Ethan Furman wrote: >>> Paul Moore also declaimed: >>>> Alternatively, you could distribute all 3 files [...] >>> >>> The problem I have with this method is that the last time I tried it >>> setup.py attempted to pre-compile all the files, resulting in syntax errors, >>> which makes for a lousy user experience. >> >> Yeah, it's harmless but ugly. I've seen that issue myself. > > For the record, I thought setting PYTHONDONTWRITEBYTECODE from inside the setup.py script: > > os.environ['PYTHONDONTWRITEBYTECODE'] = True > > might do the trick, but > > - it isn't available on 2.5 > - it doesn't work on 2.6 (didn't test on 2.7 nor 3.x) > > Oh well. Guess I'll include a note that says, "Ignore any syntax errors during 'setup.py install', they're harmless." > > :( > > -- > ~Ethan~ > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig I think the next version of pip might hide this unless an error in install actually occurs. I could be wrong though. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From nad at acm.org Tue Oct 28 21:06:57 2014 From: nad at acm.org (Ned Deily) Date: Tue, 28 Oct 2014 13:06:57 -0700 Subject: [Distutils] PEP425 - Wheel Binary Package Compatibility References: Message-ID: In article , Olivier Grisel wrote: > Homebrew-installed Python and the system Python that comes by default > on OSX are binary compatible with wheels generated with Python from > the official OSX installer from python.org. > > In current versions of pip you "just" have to rename the wheel > filenames to get them picked up from PyPI on those platforms. > > This explains the strange names of the OSX wheels for numpy for instance: > > numpy-1.9.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.w > hl > numpy-1.9.0-cp33-cp33m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64. > whl > numpy-1.9.0-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64. > whl > > https://pypi.python.org/pypi/numpy > > macosx_10_9_x86_64 is the platform tag used by Homebrew-installed > Python while macosx_10_6_intel is the platform tag of the python.org > Python. To expand on this, the platform tag is generated by Python's Distutils and wheels use and expand on the pattern established by it. For OS X, the "10_x" part is based on the minimum OS X deployment target of the Python build, normally set by the compiler -mmacosx-version-min argument or the MACOSX_DEPLOYMENT_TARGET env variable at configure time (these are standard options supported by the Apple-supplied buildtool chains). That value is saved and used by Distutils in subsequent extension module builds, although you can override it to a higher (not lower) value for a particular build. The arch values in the tag also come from the configured universal architectures. Any Python configured the same way (wrt MACOSX_DEPLOYMENT_TARGET and archs) will supply the same platform tag. So there's nothing special about Homebrew or MacPorts or python.org or the Apple-supplied system Pythons; it is assumed that any identically configured Python will provide a compatible ABI for C extension modules. We also assume that Apple will provide backward compatibility on newer releases of OS X, such that, for example, an extension module compiled for 10.6 will run just fine on later OS X releases *and* will run with a Python configured for a later OS X release, assuming there is a non-zero intersection of universal archs between the interpreter and the extension module. In practice, those assumptions seems to have worked fairly well on OS X, starting from the beginning of universal arch support and at least up to now although they may break down in the future. It's still not perfect. For example, the platform string doesn't take into account the Python 2 --enable-unicode=ucs2/ucs4 build option; but ucs4 builds aren't widely used on OS X and Python 3 no longer has this problem. (For C++ extension modules, there also may be issues with Apple's change in default c++ libraries in OS X 10.9.) On Linux systems, it can be dangerous to make similar simplifying assumptions, with many more variables in play, much less reflected in the platform string. -- Ned Deily, nad at acm.org From chris.barker at noaa.gov Tue Oct 28 20:34:02 2014 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 28 Oct 2014 12:34:02 -0700 Subject: [Distutils] a package with is a module In-Reply-To: <20141027153003.GA8729@fridge.pov.lt> References: <544E4D01.3030900@stoneleaf.us> <20141027153003.GA8729@fridge.pov.lt> Message-ID: On Mon, Oct 27, 2014 at 8:30 AM, Marius Gedminas wrote: > Many setup.py files fail if you run them when your working directory is > not the one where the setup.py resides itself. I think you can safely > rely on that implementation detail. > agreed, but if you really wanted to be sure you could use __file__ to get the path of the setup.y and go from there. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Oct 29 10:47:06 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Oct 2014 19:47:06 +1000 Subject: [Distutils] PEP425 - Wheel Binary Package Compatibility In-Reply-To: References: Message-ID: On 29 Oct 2014 01:02, "David Cournapeau" wrote: > > Practically speaking, there is no such a thing as ABI on Linux: even if you somehow managed to deal with glibc, you would then need to deal with fortran ABI, then with C++ ABI, etc... Dealing with this at the python level is simply intractable. That needs to be handled at the distribution level, or within a "sumo distribution" like Anaconda from Continuum or Canopy from Enthought (disclaimer, I work for Enthought), which makes the problem more tractable by picking up an ABI and stick with it for the full set of packages. This is the kind of direction we're also exploring for Fedora & EPEL: setting up a distro specific devpi instance so we can automatically publish distro compatible wheel files, as well as separating the distro level licencing and preliminary security review step from the step of repackaging in a language independent format. (I need to set up a curated PyPI mirror for work anyway, and since I work at Red Hat, and the Fedora community are open to the idea, we're working on it upstream rather than inside the firewall) This initial experiment with Python is a pilot - if we can get it to work in a way that seems sustainable, we'll look to offer it as a supported service in the Fedora 22 time frame, as well as opening things up for folks to pitch other language specific stacks. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From qwcode at gmail.com Wed Oct 29 19:36:59 2014 From: qwcode at gmail.com (Marcus Smith) Date: Wed, 29 Oct 2014 11:36:59 -0700 Subject: [Distutils] PEP425 - Wheel Binary Package Compatibility In-Reply-To: References: Message-ID: > This is the kind of direction we're also exploring for Fedora & EPEL: > setting up a distro specific devpi instance so we can automatically publish > distro compatible wheel files, as well as separating the distro level > licencing and preliminary security review step from the step of repackaging > in a language independent format. (I need to set up a curated PyPI mirror > for work anyway, and since I work at Red Hat, and the Fedora community are > open to the idea, we're working on it upstream rather than inside the > firewall) > Btw, I've been doing this at my company, that is maintaining separate wheel indexes per platform we need (in our case, cent5 and cent6). One problem here is that it's possible for pip download caches to get "corrupted" with the wrong distributions (i.e. the incompatible ones), since the distributions are not distinguishable by name Would Fedora really consider releasing a public service where there fedora release is not represented in the distribution name? or would you be wanting to get more tagging spec'd out in Wheel 2.0 first? P.S. In looking at the PEP425/427 specs again, I'm slightly confused about the "build tag". It's barely covered, but I can imagine using that possibly to distinguish dists, assuming pip support was added, alhough it's supposed to start with a number, which doesn't seem to fit the use case we'd want. -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Wed Oct 29 20:52:39 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Oct 2014 15:52:39 -0400 Subject: [Distutils] PEP425 - Wheel Binary Package Compatibility In-Reply-To: References: Message-ID: <4A12B6FF-31D9-48E9-93F8-79BE1EA87B29@stufft.io> > On Oct 29, 2014, at 2:36 PM, Marcus Smith wrote: > > > This is the kind of direction we're also exploring for Fedora & EPEL: setting up a distro specific devpi instance so we can automatically publish distro compatible wheel files, as well as separating the distro level licencing and preliminary security review step from the step of repackaging in a language independent format. (I need to set up a curated PyPI mirror for work anyway, and since I work at Red Hat, and the Fedora community are open to the idea, we're working on it upstream rather than inside the firewall) > > > Btw, I've been doing this at my company, that is maintaining separate wheel indexes per platform we need (in our case, cent5 and cent6). > One problem here is that it's possible for pip download caches to get "corrupted" with the wrong distributions (i.e. the incompatible ones), since the distributions are not distinguishable by name > Would Fedora really consider releasing a public service where there fedora release is not represented in the distribution name? or would you be wanting to get more tagging spec'd out in Wheel 2.0 first? > > P.S. In looking at the PEP425/427 specs again, I'm slightly confused about the "build tag". It's barely covered, but I can imagine using that possibly to distinguish dists, assuming pip support was added, alhough it's supposed to start with a number, which doesn't seem to fit the use case we'd want. > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig It?s a build number, it?s used incase you need to rebuild a Wheel using the same source files. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Wed Oct 29 21:46:46 2014 From: dholth at gmail.com (Daniel Holth) Date: Wed, 29 Oct 2014 16:46:46 -0400 Subject: [Distutils] PEP425 - Wheel Binary Package Compatibility In-Reply-To: <4A12B6FF-31D9-48E9-93F8-79BE1EA87B29@stufft.io> References: <4A12B6FF-31D9-48E9-93F8-79BE1EA87B29@stufft.io> Message-ID: I've always thought people might like to have a custom platform tag or otherwise customize the supported list in a configuration file loaded by pep425tags. Then you could create wheels called somewheel-4.0-py33-none-30caa8a209d6.whl (choose your own random string). Only a pip with a matching config would pay any attention to those wheels. That way we could help people manage their internal deployments without falsely advertising that we've solved the binary incompatibility feature baked into every Linux distribution. On Wed, Oct 29, 2014 at 3:52 PM, Donald Stufft wrote: > > On Oct 29, 2014, at 2:36 PM, Marcus Smith wrote: > > >> This is the kind of direction we're also exploring for Fedora & EPEL: >> setting up a distro specific devpi instance so we can automatically publish >> distro compatible wheel files, as well as separating the distro level >> licencing and preliminary security review step from the step of repackaging >> in a language independent format. (I need to set up a curated PyPI mirror >> for work anyway, and since I work at Red Hat, and the Fedora community are >> open to the idea, we're working on it upstream rather than inside the >> firewall) > > > Btw, I've been doing this at my company, that is maintaining separate wheel > indexes per platform we need (in our case, cent5 and cent6). > One problem here is that it's possible for pip download caches to get > "corrupted" with the wrong distributions (i.e. the incompatible ones), since > the distributions are not distinguishable by name > Would Fedora really consider releasing a public service where there fedora > release is not represented in the distribution name? or would you be wanting > to get more tagging spec'd out in Wheel 2.0 first? > > P.S. In looking at the PEP425/427 specs again, I'm slightly confused about > the "build tag". It's barely covered, but I can imagine using that possibly > to distinguish dists, assuming pip support was added, alhough it's supposed > to start with a number, which doesn't seem to fit the use case we'd want. > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > > It?s a build number, it?s used incase you need to rebuild a Wheel using the > same source files. > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > From qwcode at gmail.com Wed Oct 29 21:53:19 2014 From: qwcode at gmail.com (Marcus Smith) Date: Wed, 29 Oct 2014 13:53:19 -0700 Subject: [Distutils] PEP425 - Wheel Binary Package Compatibility In-Reply-To: References: <4A12B6FF-31D9-48E9-93F8-79BE1EA87B29@stufft.io> Message-ID: yes, I'm partial to a solution like this prior to wheel 2.0 (that I imagine would support additional/custom tags) On Wed, Oct 29, 2014 at 1:46 PM, Daniel Holth wrote: > I've always thought people might like to have a custom platform tag or > otherwise customize the supported list in a configuration file loaded > by pep425tags. Then you could create wheels called > somewheel-4.0-py33-none-30caa8a209d6.whl (choose your own random > string). Only a pip with a matching config would pay any attention to > those wheels. That way we could help people manage their internal > deployments without falsely advertising that we've solved the binary > incompatibility feature baked into every Linux distribution. > > On Wed, Oct 29, 2014 at 3:52 PM, Donald Stufft wrote: > > > > On Oct 29, 2014, at 2:36 PM, Marcus Smith wrote: > > > > > >> This is the kind of direction we're also exploring for Fedora & EPEL: > >> setting up a distro specific devpi instance so we can automatically > publish > >> distro compatible wheel files, as well as separating the distro level > >> licencing and preliminary security review step from the step of > repackaging > >> in a language independent format. (I need to set up a curated PyPI > mirror > >> for work anyway, and since I work at Red Hat, and the Fedora community > are > >> open to the idea, we're working on it upstream rather than inside the > >> firewall) > > > > > > Btw, I've been doing this at my company, that is maintaining separate > wheel > > indexes per platform we need (in our case, cent5 and cent6). > > One problem here is that it's possible for pip download caches to get > > "corrupted" with the wrong distributions (i.e. the incompatible ones), > since > > the distributions are not distinguishable by name > > Would Fedora really consider releasing a public service where there > fedora > > release is not represented in the distribution name? or would you be > wanting > > to get more tagging spec'd out in Wheel 2.0 first? > > > > P.S. In looking at the PEP425/427 specs again, I'm slightly confused > about > > the "build tag". It's barely covered, but I can imagine using that > possibly > > to distinguish dists, assuming pip support was added, alhough it's > supposed > > to start with a number, which doesn't seem to fit the use case we'd want. > > > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > > > > > It?s a build number, it?s used incase you need to rebuild a Wheel using > the > > same source files. > > > > --- > > Donald Stufft > > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Oct 30 00:57:03 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 30 Oct 2014 09:57:03 +1000 Subject: [Distutils] PEP425 - Wheel Binary Package Compatibility In-Reply-To: References: <4A12B6FF-31D9-48E9-93F8-79BE1EA87B29@stufft.io> Message-ID: On 30 Oct 2014 07:20, "Marcus Smith" wrote: > > yes, I'm partial to a solution like this prior to wheel 2.0 (that I imagine would support additional/custom tags) +1 for being able to add additional custom platform tags in the file naming convention from me as well. As Marcus noted earlier, even if you set up distro specific indexes currently, there's nothing built into the tooling to keep you from trying to install (e.g.) a Fedora 21 wheel on RHEL or CentOS 5 (which is highly unlikely to work, given that the core ABIs in RHEL/CentOS 5 are 7 or 8 years old at this point). We'd be highly unlikely to flip the switch from "experimental service, use at your own risk" to "fully supported Fedora feature" while that's still the case. With arbitrary platform tags, we could inject that into the wheel filenames as part of the build process, and then again when invoking pip. That opens things up for us to figure out how to best flag compatibility on the distro side, without committing to a specific approach upstream (not yet, anyway). An alternative approach would be to add an "additional wheel suffix" setting for pip that allowed us to have names with endings like ".fc21.whl" or ".el7.whl" recognised as valid wheel files. I'm not that worried about the exact details though - the main feature I'd like is the ability to create wheel files that pip will ignore by default, but will accept if I specifically tell it what to look for. Cheers, Nick. > > On Wed, Oct 29, 2014 at 1:46 PM, Daniel Holth wrote: >> >> I've always thought people might like to have a custom platform tag or >> otherwise customize the supported list in a configuration file loaded >> by pep425tags. Then you could create wheels called >> somewheel-4.0-py33-none-30caa8a209d6.whl (choose your own random >> string). Only a pip with a matching config would pay any attention to >> those wheels. That way we could help people manage their internal >> deployments without falsely advertising that we've solved the binary >> incompatibility feature baked into every Linux distribution. >> >> On Wed, Oct 29, 2014 at 3:52 PM, Donald Stufft wrote: >> > >> > On Oct 29, 2014, at 2:36 PM, Marcus Smith wrote: >> > >> > >> >> This is the kind of direction we're also exploring for Fedora & EPEL: >> >> setting up a distro specific devpi instance so we can automatically publish >> >> distro compatible wheel files, as well as separating the distro level >> >> licencing and preliminary security review step from the step of repackaging >> >> in a language independent format. (I need to set up a curated PyPI mirror >> >> for work anyway, and since I work at Red Hat, and the Fedora community are >> >> open to the idea, we're working on it upstream rather than inside the >> >> firewall) >> > >> > >> > Btw, I've been doing this at my company, that is maintaining separate wheel >> > indexes per platform we need (in our case, cent5 and cent6). >> > One problem here is that it's possible for pip download caches to get >> > "corrupted" with the wrong distributions (i.e. the incompatible ones), since >> > the distributions are not distinguishable by name >> > Would Fedora really consider releasing a public service where there fedora >> > release is not represented in the distribution name? or would you be wanting >> > to get more tagging spec'd out in Wheel 2.0 first? >> > >> > P.S. In looking at the PEP425/427 specs again, I'm slightly confused about >> > the "build tag". It's barely covered, but I can imagine using that possibly >> > to distinguish dists, assuming pip support was added, alhough it's supposed >> > to start with a number, which doesn't seem to fit the use case we'd want. >> > >> > >> > _______________________________________________ >> > Distutils-SIG maillist - Distutils-SIG at python.org >> > https://mail.python.org/mailman/listinfo/distutils-sig >> > >> > >> > It?s a build number, it?s used incase you need to rebuild a Wheel using the >> > same source files. >> > >> > --- >> > Donald Stufft >> > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >> > >> > >> > _______________________________________________ >> > Distutils-SIG maillist - Distutils-SIG at python.org >> > https://mail.python.org/mailman/listinfo/distutils-sig >> > > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Oct 30 01:17:16 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 29 Oct 2014 20:17:16 -0400 Subject: [Distutils] PEP425 - Wheel Binary Package Compatibility In-Reply-To: References: <4A12B6FF-31D9-48E9-93F8-79BE1EA87B29@stufft.io> Message-ID: <082D8B0C-59AA-4EB4-B0F5-315A0B03C2E4@stufft.io> > On Oct 29, 2014, at 7:57 PM, Nick Coghlan wrote: > > > On 30 Oct 2014 07:20, "Marcus Smith" > wrote: > > > > yes, I'm partial to a solution like this prior to wheel 2.0 (that I imagine would support additional/custom tags) > > +1 for being able to add additional custom platform tags in the file naming convention from me as well. As Marcus noted earlier, even if you set up distro specific indexes currently, there's nothing built into the tooling to keep you from trying to install (e.g.) a Fedora 21 wheel on RHEL or CentOS 5 (which is highly unlikely to work, given that the core ABIs in RHEL/CentOS 5 are 7 or 8 years old at this point). > > We'd be highly unlikely to flip the switch from "experimental service, use at your own risk" to "fully supported Fedora feature" while that's still the case. > > With arbitrary platform tags, we could inject that into the wheel filenames as part of the build process, and then again when invoking pip. > > That opens things up for us to figure out how to best flag compatibility on the distro side, without committing to a specific approach upstream (not yet, anyway). > > An alternative approach would be to add an "additional wheel suffix" setting for pip that allowed us to have names with endings like ".fc21.whl" or ".el7.whl" recognised as valid wheel files. > > I'm not that worried about the exact details though - the main feature I'd like is the ability to create wheel files that pip will ignore by default, but will accept if I specifically tell it what to look for. > > Do we plan for this to be allowed to upload to PyPI proper? These custom tags? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Thu Oct 30 02:25:27 2014 From: dholth at gmail.com (Daniel Holth) Date: Wed, 29 Oct 2014 21:25:27 -0400 Subject: [Distutils] PEP425 - Wheel Binary Package Compatibility In-Reply-To: <082D8B0C-59AA-4EB4-B0F5-315A0B03C2E4@stufft.io> References: <4A12B6FF-31D9-48E9-93F8-79BE1EA87B29@stufft.io> <082D8B0C-59AA-4EB4-B0F5-315A0B03C2E4@stufft.io> Message-ID: On Wed, Oct 29, 2014 at 8:17 PM, Donald Stufft wrote: > > On Oct 29, 2014, at 7:57 PM, Nick Coghlan wrote: > > > On 30 Oct 2014 07:20, "Marcus Smith" wrote: >> >> yes, I'm partial to a solution like this prior to wheel 2.0 (that I >> imagine would support additional/custom tags) > > +1 for being able to add additional custom platform tags in the file naming > convention from me as well. As Marcus noted earlier, even if you set up > distro specific indexes currently, there's nothing built into the tooling to > keep you from trying to install (e.g.) a Fedora 21 wheel on RHEL or CentOS 5 > (which is highly unlikely to work, given that the core ABIs in RHEL/CentOS 5 > are 7 or 8 years old at this point). > > We'd be highly unlikely to flip the switch from "experimental service, use > at your own risk" to "fully supported Fedora feature" while that's still the > case. > > With arbitrary platform tags, we could inject that into the wheel filenames > as part of the build process, and then again when invoking pip. > > That opens things up for us to figure out how to best flag compatibility on > the distro side, without committing to a specific approach upstream (not > yet, anyway). > > An alternative approach would be to add an "additional wheel suffix" setting > for pip that allowed us to have names with endings like ".fc21.whl" or > ".el7.whl" recognised as valid wheel files. > > I'm not that worried about the exact details though - the main feature I'd > like is the ability to create wheel files that pip will ignore by default, > but will accept if I specifically tell it what to look for. > > > > Do we plan for this to be allowed to upload to PyPI proper? These custom > tags? No From richard at python.org Thu Oct 30 01:29:35 2014 From: richard at python.org (Richard Jones) Date: Thu, 30 Oct 2014 11:29:35 +1100 Subject: [Distutils] PEP425 - Wheel Binary Package Compatibility In-Reply-To: <082D8B0C-59AA-4EB4-B0F5-315A0B03C2E4@stufft.io> References: <4A12B6FF-31D9-48E9-93F8-79BE1EA87B29@stufft.io> <082D8B0C-59AA-4EB4-B0F5-315A0B03C2E4@stufft.io> Message-ID: It could get quite messy, without consensus. I figured custom tags would be a more local thing. On 30 October 2014 11:17, Donald Stufft wrote: > > On Oct 29, 2014, at 7:57 PM, Nick Coghlan wrote: > > > On 30 Oct 2014 07:20, "Marcus Smith" wrote: > > > > yes, I'm partial to a solution like this prior to wheel 2.0 (that I > imagine would support additional/custom tags) > > +1 for being able to add additional custom platform tags in the file > naming convention from me as well. As Marcus noted earlier, even if you set > up distro specific indexes currently, there's nothing built into the > tooling to keep you from trying to install (e.g.) a Fedora 21 wheel on RHEL > or CentOS 5 (which is highly unlikely to work, given that the core ABIs in > RHEL/CentOS 5 are 7 or 8 years old at this point). > > We'd be highly unlikely to flip the switch from "experimental service, use > at your own risk" to "fully supported Fedora feature" while that's still > the case. > > With arbitrary platform tags, we could inject that into the wheel > filenames as part of the build process, and then again when invoking pip. > > That opens things up for us to figure out how to best flag compatibility > on the distro side, without committing to a specific approach upstream (not > yet, anyway). > > An alternative approach would be to add an "additional wheel suffix" > setting for pip that allowed us to have names with endings like ".fc21.whl" > or ".el7.whl" recognised as valid wheel files. > > I'm not that worried about the exact details though - the main feature I'd > like is the ability to create wheel files that pip will ignore by default, > but will accept if I specifically tell it what to look for. > > > Do we plan for this to be allowed to upload to PyPI proper? These custom > tags? > > --- > Donald Stufft > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Oct 30 16:18:57 2014 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 31 Oct 2014 01:18:57 +1000 Subject: [Distutils] PEP425 - Wheel Binary Package Compatibility In-Reply-To: References: <4A12B6FF-31D9-48E9-93F8-79BE1EA87B29@stufft.io> <082D8B0C-59AA-4EB4-B0F5-315A0B03C2E4@stufft.io> Message-ID: On 30 October 2014 10:29, Richard Jones wrote: > It could get quite messy, without consensus. I figured custom tags would be > a more local thing. Yep, that's what I would suggest - keep it off PyPI for now, but provide some way to actually try the concept in real environments. That way future standardisation can be based on experience rather than trying to guess potential use cases in advance. Cheers, Nick. > > On 30 October 2014 11:17, Donald Stufft wrote: >> >> >> On Oct 29, 2014, at 7:57 PM, Nick Coghlan wrote: >> >> >> On 30 Oct 2014 07:20, "Marcus Smith" wrote: >> > >> > yes, I'm partial to a solution like this prior to wheel 2.0 (that I >> > imagine would support additional/custom tags) >> >> +1 for being able to add additional custom platform tags in the file >> naming convention from me as well. As Marcus noted earlier, even if you set >> up distro specific indexes currently, there's nothing built into the tooling >> to keep you from trying to install (e.g.) a Fedora 21 wheel on RHEL or >> CentOS 5 (which is highly unlikely to work, given that the core ABIs in >> RHEL/CentOS 5 are 7 or 8 years old at this point). >> >> We'd be highly unlikely to flip the switch from "experimental service, use >> at your own risk" to "fully supported Fedora feature" while that's still the >> case. >> >> With arbitrary platform tags, we could inject that into the wheel >> filenames as part of the build process, and then again when invoking pip. >> >> That opens things up for us to figure out how to best flag compatibility >> on the distro side, without committing to a specific approach upstream (not >> yet, anyway). >> >> An alternative approach would be to add an "additional wheel suffix" >> setting for pip that allowed us to have names with endings like ".fc21.whl" >> or ".el7.whl" recognised as valid wheel files. >> >> I'm not that worried about the exact details though - the main feature I'd >> like is the ability to create wheel files that pip will ignore by default, >> but will accept if I specifically tell it what to look for. >> >> >> >> Do we plan for this to be allowed to upload to PyPI proper? These custom >> tags? >> >> --- >> Donald Stufft >> PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> > -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From robertc at robertcollins.net Fri Oct 31 03:24:55 2014 From: robertc at robertcollins.net (Robert Collins) Date: Fri, 31 Oct 2014 15:24:55 +1300 Subject: [Distutils] bandersnatch failing on MovieMacro? Message-ID: 2014-10-31 12:01:05,368 ERROR: Error syncing package: MovieMacro at 1288231 Traceback (most recent call last): File "/home/robertc/.virtualenvs/bandersnatch/local/lib/python2.7/site-packages/bandersnatch/pa ckage.py", line 63, in sync self.sync_simple_page() File "/home/robertc/.virtualenvs/bandersnatch/local/lib/python2.7/site-packages/bandersnatch/pa ckage.py", line 106, in sync_simple_page '/simple/{0}/'.format(self.quoted_name), self.serial) File "/home/robertc/.virtualenvs/bandersnatch/local/lib/python2.7/site-packages/bandersnatch/ma ster.py", line 44, in get r.raise_for_status() File "/home/robertc/.virtualenvs/bandersnatch/local/lib/python2.7/site-packages/requests/models .py", line 773, in raise_for_status raise HTTPError(http_error_msg, response=self) HTTPError: 404 Client Error: Not Found 2014-10-31 12:01:05,382 INFO: Generating global index page. This is aborting mirror syncs causing new packages etc not to propogate. Is this a bandersnatch bug (e.g. it should keep trying othe things) or server side issue? -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From reinout at vanrees.org Fri Oct 31 12:38:14 2014 From: reinout at vanrees.org (Reinout van Rees) Date: Fri, 31 Oct 2014 12:38:14 +0100 Subject: [Distutils] buildout 2.2.3: not completely right? Message-ID: Hi, Since buildout 2.2.3 is out, I've been getting mails out of quite some jenkins jobs. Those jobs always run "python bootstrap.py" followed by "bin/buildout". We always pin buildout, so most of them are at version 2.2.1 $ python bootstrap.py Setting socket time out to 1 seconds. Generated script ' $ bin/buildout Setting socket time out to 1 seconds. While: Installing. Loading extensions. Error: There is a version conflict. We already have: zc.buildout 2.2.3 "python bootstrap.py" generates a bin/buildout with a 2.2.3 version in it. And running bin/buildout doesn't want to downgrade to the specified version 2.2.1. The bootstrap.py I'm using is http://downloads.buildout.org/2/bootstrap.py I've seen some other problems with versions when there's a globally installed version already available at a higher version. A colleague installed ansible globally on the jenkins machine (...) so there's a new jinja2 in usr/local/lib/pythonsomething. The jinja2 in buildout is pinned to a lower version, but buildout refuses to downgrade it and fails with an "I already have that new version". So... I haven't totally tracked it down yet, but I'm throwing it out on the list here in case others have seen it too. I'll continue debugging after lunch :-) Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "Learning history by destroying artifacts is a time-honored atrocity" From reinout at vanrees.org Fri Oct 31 14:21:49 2014 From: reinout at vanrees.org (Reinout van Rees) Date: Fri, 31 Oct 2014 14:21:49 +0100 Subject: [Distutils] buildout 2.2.3: not completely right? In-Reply-To: References: Message-ID: <54538CED.7070205@vanrees.org> On 31-10-14 12:38, Reinout van Rees wrote: > > $ python bootstrap.py > Setting socket time out to 1 seconds. > Generated script > ' > $ bin/buildout > Setting socket time out to 1 seconds. > While: > Installing. > Loading extensions. > Error: There is a version conflict. > We already have: zc.buildout 2.2.3 Ok, I tracked it down: https://github.com/buildout/buildout/issues/198#issuecomment-61258074 The short summary: the main problem is line 652: https://github.com/buildout/buildout/pull/194/files#diff-784f9bb654bbcee96dcbc547bbddd493R652 The working set is queried for just the package name. It grabs whatever it can (a system package? a zc.buildout 2.2.3 installed by the bootstrap.py) and uses that. It doesn't look at the actual requirement, just at the plain package name (req.key). Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "Learning history by destroying artifacts is a time-honored atrocity" From reinout at vanrees.org Fri Oct 31 18:16:22 2014 From: reinout at vanrees.org (Reinout van Rees) Date: Fri, 31 Oct 2014 18:16:22 +0100 Subject: [Distutils] buildout 2.2.3: not completely right? In-Reply-To: <54538CED.7070205@vanrees.org> References: <54538CED.7070205@vanrees.org> Message-ID: On 31-10-14 14:21, Reinout van Rees wrote: > Ok, I tracked it down: > https://github.com/buildout/buildout/issues/198#issuecomment-61258074 And I have a pull request that fixes it. https://github.com/buildout/buildout/pull/199 All the buildouts that failed this morning work again with this fix. Reinout -- Reinout van Rees http://reinout.vanrees.org/ reinout at vanrees.org http://www.nelen-schuurmans.nl/ "Learning history by destroying artifacts is a time-honored atrocity"