But I'd really prefer to just grab a block of JSON-LD from the top of the page (and/or a attribute). >From https://en.wikipedia.org/wiki/Schema.org#JSON-LD : ```html ``` > > > If we were designing this from scratch I might agree with you, though I > think > one of the biggest benefits of the "simple" repository is that it's > extremely > simple for someone to get started with it, a file system and basically any > web server is enough. So I'm not sure it's worth it to add the explicit rel > even if we were going from scratch if we lose that property. > > > > > Hmm, that's convention based and does not allow detecting > > the presence of such signatures without actually trying a download. > > > > I think it would be better to make the availability explicit > > by adding an attribute to the link element (just like for other > > such features). > > I'm OK with adding the attribute to links, though we should still mandate > the > location. Neither pip nor setuptools will do anything with the PGP > signatures > but some other tooling might. The legacy behavior of "just try the link" > will > still work then, and if someone wants to do it more efficiently the > attribute > is there. I'm not sure it's going to be generally useful since the signing > on > PyPI doesn't really have a coherent threat model so it doesn't really > protect > against much. > > > > > A dynamic API can be added as addition, but is hardly ever required > > for installation. > > I think I've misled you! > > I have no plans to make the installer API dynamic. I want as many requests > as > possible to be served out of the Fastly cache instead of hitting the > backends > as possible which means whatever we end up with will focus on static > responses > and expecting the clients to do more requests and handle more things on the > client side rather than just querying the server for it. > OT, indeed, but for/against downloading the *entire* package catalog (like yum,conda,dnf)? > > However, I don't think that HTML is a very good data serialization format, > it's > primary purpose is for document markup which isn't really the same thing as > data serialization. One such shortcoming is the total lack of any real data > type except for strings, looking at just the "has gpg signature" thing from > above, it'd be nicer if that could be modeled as: > > [ > {"url": "/path/to/filename.tar.gz", "has_sig": True} > ] > > Instead of relying on the presence or absence of a value (which doens't > work > if you need to detect the difference between False/None) or using special > values that can be converted to some other data type if you know ahead of > time > you need to do that for that particular value. > > I don't know for a fact it'll be JSON, but I think it's an obvious choice > given > that it's in the standard library, it's a reasonable data serialization > format, > and it's human readable. When it comes to that time I think it'd be > reasonable > to explore other formats to see if they make sense too, but whatever it > will > be it's my intent it'll be static. > JSON-LD is/should be parseable without a JSON-LD library (the @context (which can be implicit or explicit) is just ignored) > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nate at bx.psu.edu Tue Sep 8 19:29:29 2015 From: nate at bx.psu.edu (Nate Coraor) Date: Tue, 8 Sep 2015 13:29:29 -0400 Subject: [Distutils] Working toward Linux wheel support In-Reply-To: References:

Message-ID: On Mon, Sep 7, 2015 at 12:02 PM, Donald Stufft wrote: > On September 3, 2015 at 1:23:03 PM, Nate Coraor (nate at bx.psu.edu) wrote: > > >>> > > >>> I'll create PRs for this against wheel and pip shortly. I can also > work > > >>> on a PEP for the platform tag - I don't think it's going to need to > be a > > >>> big one. Are there any preferences as to whether this should be a > new PEP > > >>> or an update to 425? > > >>> > > Coming back to this, I'm wondering if we should include the libc > implementation/version in a less generic, but still generic linux wheel. > Right > now if you staticly link I think the only platform ABIs you need to worry > about > are libc and Python itself. Python itself is handled already but libc is > not. The only thing I've seen so far is "build on an old enough version of glibc > that it handles anything sane", but not all versions of Linux even use > glibc at > all. This proposal makes a lot of sense to me. pip will need an update to do the backwards compatibility, and it may be a little ugly to do this all on the platform tag. For example, linux_x86_64_ubuntu_12_04 wheels should not be installed on systems that identify as linux_x86_64_ubuntu_14_04, but linux_x86_64_glibc_2_15 wheels can be installed on systems that identify as linux_x86_64_glibc_2_19. pip would need to maintain a list of which tag prefixes or patterns should be considered backward compatible, and which should not. Granted, new libcs do not pop up overnight, so it's not exactly a nightmare scenario. Wheel should be updated to generate the "libc-generic" wheels by default when nothing other than libc is dynamically linked. It'll need libc vendor/version detection. Alternatively, the platform tag could be split in two, in which case you have a "generic" portion (which would probably be what it currently is, distutils.util.get_platform()) and a "specific" portion (the distro or libc), possibly prefixed with something to avoid having to maintain a list of what's version compatible and what's not, (e.g. 'd_ubuntu_14_04' vs. 'c_glibc_2_19')? I don't think there is a strong case to include the libc version in the specific portion when a distro version will also be specified, because the distro is supposed to define the ABI (at least in the case of distros with stable ABIs), and that includes the libc compatibility. So for psycopg2 wheels you'd get a "distro" wheel (linux_x86_64-d_ubuntu_14_04) but for SQLAlchemy, you'd get a "libc-generic" wheel (linux_x86_64-c_glibc_2_19). It's then up to PyPI project owners to build on whatever platforms they wish to support. --nate > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Sep 8 20:33:05 2015 From: donald at stufft.io (Donald Stufft) Date: Tue, 8 Sep 2015 14:33:05 -0400 Subject: [Distutils] Working toward Linux wheel support In-Reply-To: References:

Message-ID: On September 8, 2015 at 1:29:53 PM, Nate Coraor (nate at bx.psu.edu) wrote: > On Mon, Sep 7, 2015 at 12:02 PM, Donald Stufft wrote: > > > On September 3, 2015 at 1:23:03 PM, Nate Coraor (nate at bx.psu.edu) wrote: > > > >>> > > > >>> I'll create PRs for this against wheel and pip shortly. I can also > > work > > > >>> on a PEP for the platform tag - I don't think it's going to need to > > be a > > > >>> big one. Are there any preferences as to whether this should be a > > new PEP > > > >>> or an update to 425? > > > >>> > > > > Coming back to this, I'm wondering if we should include the libc > > implementation/version in a less generic, but still generic linux wheel. > > Right > > now if you staticly link I think the only platform ABIs you need to worry > > about > > are libc and Python itself. Python itself is handled already but libc is > > not. > > The only thing I've seen so far is "build on an old enough version of glibc > > that it handles anything sane", but not all versions of Linux even use > > glibc at > > all. > > > This proposal makes a lot of sense to me. pip will need an update to do the > backwards compatibility, and it may be a little ugly to do this all on the > platform tag. For example, linux_x86_64_ubuntu_12_04 wheels should not be > installed on systems that identify as linux_x86_64_ubuntu_14_04, but > linux_x86_64_glibc_2_15 wheels can be installed on systems that identify as > linux_x86_64_glibc_2_19. pip would need to maintain a list of which tag > prefixes or patterns should be considered backward compatible, and which > should not. Granted, new libcs do not pop up overnight, so it's not exactly > a nightmare scenario. > > Wheel should be updated to generate the "libc-generic" wheels by default > when nothing other than libc is dynamically linked. It'll need libc > vendor/version detection. > > Alternatively, the platform tag could be split in two, in which case you > have a "generic" portion (which would probably be what it currently is, > distutils.util.get_platform()) and a "specific" portion (the distro or > libc), possibly prefixed with something to avoid having to maintain a list > of what's version compatible and what's not, (e.g. 'd_ubuntu_14_04' vs. > 'c_glibc_2_19')? > > I don't think there is a strong case to include the libc version in the > specific portion when a distro version will also be specified, because the > distro is supposed to define the ABI (at least in the case of distros with > stable ABIs), and that includes the libc compatibility. So for psycopg2 > wheels you'd get a "distro" wheel (linux_x86_64-d_ubuntu_14_04) but for > SQLAlchemy, you'd get a "libc-generic" wheel (linux_x86_64-c_glibc_2_19). > > It's then up to PyPI project owners to build on whatever platforms they > wish to support. > I think it's reasonable to not include the libc when the wheel is distro specific. I think the barrier to entry on adding new tags is far lower than adding a whole new type of tag. Right now, I think our longest tag is for OSX which is something like macosx_10_10_x86_64 at 19 chars, I don't think it's much worse to have something like linux_glibc_2_19_x86_64 at 23 chars, or linux_ubuntu_14_04_x86_64 at 25 chars. I don't think we need the special c or d prefix, we can just treat it as ==, and special case glibc as >= like we're currently special casing the macosx wheels to be >=. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From wes.turner at gmail.com Tue Sep 8 21:14:53 2015 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 8 Sep 2015 14:14:53 -0500 Subject: [Distutils] Working toward Linux wheel support In-Reply-To: References:

Message-ID: On Sep 8, 2015 1:33 PM, "Donald Stufft" wrote: > > On September 8, 2015 at 1:29:53 PM, Nate Coraor (nate at bx.psu.edu) wrote: > > On Mon, Sep 7, 2015 at 12:02 PM, Donald Stufft wrote: > > > > > On September 3, 2015 at 1:23:03 PM, Nate Coraor (nate at bx.psu.edu) wrote: > > > > >>> > > > > >>> I'll create PRs for this against wheel and pip shortly. I can also > > > work > > > > >>> on a PEP for the platform tag - I don't think it's going to need to > > > be a > > > > >>> big one. Are there any preferences as to whether this should be a > > > new PEP > > > > >>> or an update to 425? > > > > >>> > > > > > > Coming back to this, I'm wondering if we should include the libc > > > implementation/version in a less generic, but still generic linux wheel. > > > Right > > > now if you staticly link I think the only platform ABIs you need to worry > > > about > > > are libc and Python itself. Python itself is handled already but libc is > > > not. > > > > The only thing I've seen so far is "build on an old enough version of glibc > > > that it handles anything sane", but not all versions of Linux even use > > > glibc at > > > all. > > > > > > This proposal makes a lot of sense to me. pip will need an update to do the > > backwards compatibility, and it may be a little ugly to do this all on the > > platform tag. For example, linux_x86_64_ubuntu_12_04 wheels should not be > > installed on systems that identify as linux_x86_64_ubuntu_14_04, but > > linux_x86_64_glibc_2_15 wheels can be installed on systems that identify as > > linux_x86_64_glibc_2_19. pip would need to maintain a list of which tag > > prefixes or patterns should be considered backward compatible, and which > > should not. Granted, new libcs do not pop up overnight, so it's not exactly > > a nightmare scenario. Could there be shim packages here? How is this a different dependency? > > > > Wheel should be updated to generate the "libc-generic" wheels by default > > when nothing other than libc is dynamically linked. It'll need libc > > vendor/version detection. > > > > Alternatively, the platform tag could be split in two, in which case you > > have a "generic" portion (which would probably be what it currently is, > > distutils.util.get_platform()) and a "specific" portion (the distro or > > libc), possibly prefixed with something to avoid having to maintain a list > > of what's version compatible and what's not, (e.g. 'd_ubuntu_14_04' vs. > > 'c_glibc_2_19')? > > > > I don't think there is a strong case to include the libc version in the > > specific portion when a distro version will also be specified, because the > > distro is supposed to define the ABI (at least in the case of distros with > > stable ABIs), and that includes the libc compatibility. So for psycopg2 > > wheels you'd get a "distro" wheel (linux_x86_64-d_ubuntu_14_04) but for > > SQLAlchemy, you'd get a "libc-generic" wheel (linux_x86_64-c_glibc_2_19). > > > > It's then up to PyPI project owners to build on whatever platforms they > > wish to support. > > > > I think it's reasonable to not include the libc when the wheel is distro > specific. I think the barrier to entry on adding new tags is far lower than > adding a whole new type of tag. Right now, I think our longest tag is for OSX > which is something like macosx_10_10_x86_64 at 19 chars, I don't think it's > much worse to have something like linux_glibc_2_19_x86_64 at 23 chars, or > linux_ubuntu_14_04_x86_64 at 25 chars. I don't think we need the special c or > d prefix, we can just treat it as ==, and special case glibc as >= like we're > currently special casing the macosx wheels to be >=. > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From leorochael at gmail.com Tue Sep 8 21:18:10 2015 From: leorochael at gmail.com (Leonardo Rochael Almeida) Date: Tue, 8 Sep 2015 16:18:10 -0300 Subject: [Distutils] Working toward Linux wheel support In-Reply-To: References:

Message-ID: Hi, Going back in time to this old post, but I think it becomes more relevant now that Nate's work is being completed: On 13 August 2015 at 22:47, Nathaniel Smith wrote: > On Thu, Aug 13, 2015 at 12:30 PM, Leonardo Rochael Almeida > wrote: > > > > On 13 August 2015 at 11:07, Nate Coraor wrote: > >> > >> On Wed, Aug 12, 2015 at 9:05 PM, Nathaniel Smith wrote: > >>> > [...] > >>> (2) the special hard-coded tag "centos5". (That's what everyone > actually > >>> uses in practice, right?) > >> > >> > >> The idea here is that we should attempt to install centos5 wheels if > more > >> specific wheels for the platform aren't available? > > > > > > Just my opinion, but although I'm +1 on Nate's efforts, I'm -1 on both > the > > standard behavior for installation being the exact platform tag, and an > > automatic fallback to cento5. > > > > IMO, on Linux, the default should always be to opt in to the desired > > platform tags. > > > > We could make it so that the word `default` inside > > `binary-compatibility.cfg` means an exact match on the distro version, so > > that we could simplify the documentation. > > > > But I don't want to upgrade to pip and suddenly find myself installing > > binary wheels compiled by whomever for whatever platform I have no > control > > with, even assuming the best of the package builders intentions. > > > > And I certainly don't want centos5 wheels accidentally installed on my > > ubuntu servers unless I very specifically asked for them. > > > > The tiny pain inflicted by telling users to add a one-line text file in a > > very well known location (or two lines, for the added centos5), so that > they > > can get the benefit of binary wheels on linux, is very small compared to > the > > pain of repeatable install scripts suddenly behaving differently and > > installing binary wheels in systems that were prepared to pay the price > of > > source installs, including the setting of build environment variables > that > > correctly tweaked their build process. > > I think there are two issues here: > > 1) You don't want centos5 wheels "accidentally" installed on an ubuntu > server: Fair enough, you're right; we should probably make the "this > wheel should work on pretty much any linux out there" tag be something > that distributors have to explicitly opt into (similar to how they > have to opt into creating universal wheels), rather than having it be > something you could get by just typing 'pip wheel foo' on the right > (wrong) machine. > I agree that generating something like "this linux binary wheel is generically installable" should be opt-in, yes. But I also feel strongly that installing such a generic wheel should also be opt in. I guess that if we go into the direction of being able to generate wheels with a libc tag rather than a distro tag, like Nate and Donald are now discussing, then we could get both kinds of opt-in by specifying the libc tag in `binary-compatibility.cfg`. > 2) You want it to be the case that if I type 'pip install foo' on a > Linux machine, and pip finds both an sdist and a wheel, where the > wheel is definitely compatible with the current system, then it should > still always prefer the sdist unless configured otherwise: Here I > disagree strongly. This is inconsistent with how things work on every > other platform, it's inconsistent with how pip is being used on Linux > right now with private wheelhouses, and the "tiny pain" of editing a > file in /etc is a huge barrier to new users, many of whom are > uncomfortable editing config files and may not have root access. Not having root access shouldn't be an issue, as there should be a user-level and virtualenv level equivalents to a `binary-compatibility.cfg` on `/etc`, and perhaps it could even be included in `requirements.txt` for a project, so users of a project might not even have to bother setting up `binary-compatibility.cfg`. However, you make an excellent point: not handling binary wheels on Linux by default (at least with exact platform tag matching) would mean having different behavior between Linux and Mac/Windows. Still, I wouldn't want a random binary wheel suddenly finding its way into my servers, and I would like a way to opt out of it, for "reasons" (ex. I might have special build flags, or a special compiler, or maybe I'm still waiting for TUF before trusting other peoples binaries on my servers). So I'd like to propose that the installation tooling (eg. `pip`, `distlib`) should allow the user to specify which index servers to trust for receiving binary wheels and which to trust only for pure python wheels or sdists. It could trust them all by default, to maintain the current behavior (where `all` means only pypi unless I specified more, obviously), but I'd like a switch to limit this trust to a subset of the specified index servers. Regards, Leo -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Tue Sep 8 21:22:56 2015 From: donald at stufft.io (Donald Stufft) Date: Tue, 8 Sep 2015 15:22:56 -0400 Subject: [Distutils] Working toward Linux wheel support In-Reply-To: References:

Message-ID: On September 8, 2015 at 3:21:26 PM, Leonardo Rochael Almeida (leorochael at gmail.com) wrote: > > Still, I wouldn't want a random binary wheel suddenly finding its way into > my servers, and I would like a way to opt out of it, for "reasons" (ex. I > might have special build flags, or a special compiler, or maybe I'm still > waiting for TUF before trusting other peoples binaries on my servers). > ?no-binary packages,that,have,binaries ? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From dholth at gmail.com Tue Sep 8 21:32:09 2015 From: dholth at gmail.com (Daniel Holth) Date: Tue, 08 Sep 2015 19:32:09 +0000 Subject: [Distutils] Working toward Linux wheel support In-Reply-To: References:

Message-ID: https://www.python.org/dev/peps/pep-0425/#platform-tag is currently defined in terms of distutils get_platform(). Instead, it could be defined more abstractly to read something like "The platform tag expresses which system(s) might be capable of running or linking with binary components of the package." This would express what the tag is for rather than the list of allowed values. Then a "legal" change in the list of allowed values would not necessarily be effected by changing the distutils get_platform function. As for whether a binary is allowed from a particular server the idea of using a different list of compatible/allowed tags per-package-source has floated around. Distasteful amount of configuration though. Something like the Internet Explorer security zones where you have categories of remotes... On Tue, Sep 8, 2015 at 3:14 PM Wes Turner wrote: > > On Sep 8, 2015 1:33 PM, "Donald Stufft" wrote: > > > > On September 8, 2015 at 1:29:53 PM, Nate Coraor (nate at bx.psu.edu) wrote: > > > On Mon, Sep 7, 2015 at 12:02 PM, Donald Stufft wrote: > > > > > > > On September 3, 2015 at 1:23:03 PM, Nate Coraor (nate at bx.psu.edu) > wrote: > > > > > >>> > > > > > >>> I'll create PRs for this against wheel and pip shortly. I can > also > > > > work > > > > > >>> on a PEP for the platform tag - I don't think it's going to > need to > > > > be a > > > > > >>> big one. Are there any preferences as to whether this should > be a > > > > new PEP > > > > > >>> or an update to 425? > > > > > >>> > > > > > > > > Coming back to this, I'm wondering if we should include the libc > > > > implementation/version in a less generic, but still generic linux > wheel. > > > > Right > > > > now if you staticly link I think the only platform ABIs you need to > worry > > > > about > > > > are libc and Python itself. Python itself is handled already but > libc is > > > > not. > > > > > > The only thing I've seen so far is "build on an old enough version of > glibc > > > > that it handles anything sane", but not all versions of Linux even > use > > > > glibc at > > > > all. > > > > > > > > > This proposal makes a lot of sense to me. pip will need an update to > do the > > > backwards compatibility, and it may be a little ugly to do this all on > the > > > platform tag. For example, linux_x86_64_ubuntu_12_04 wheels should not > be > > > installed on systems that identify as linux_x86_64_ubuntu_14_04, but > > > linux_x86_64_glibc_2_15 wheels can be installed on systems that > identify as > > > linux_x86_64_glibc_2_19. pip would need to maintain a list of which tag > > > prefixes or patterns should be considered backward compatible, and > which > > > should not. Granted, new libcs do not pop up overnight, so it's not > exactly > > > a nightmare scenario. > > Could there be shim packages here? > How is this a different dependency? > > > > > > > Wheel should be updated to generate the "libc-generic" wheels by > default > > > when nothing other than libc is dynamically linked. It'll need libc > > > vendor/version detection. > > > > > > Alternatively, the platform tag could be split in two, in which case > you > > > have a "generic" portion (which would probably be what it currently is, > > > distutils.util.get_platform()) and a "specific" portion (the distro or > > > libc), possibly prefixed with something to avoid having to maintain a > list > > > of what's version compatible and what's not, (e.g. 'd_ubuntu_14_04' vs. > > > 'c_glibc_2_19')? > > > > > > I don't think there is a strong case to include the libc version in the > > > specific portion when a distro version will also be specified, because > the > > > distro is supposed to define the ABI (at least in the case of distros > with > > > stable ABIs), and that includes the libc compatibility. So for psycopg2 > > > wheels you'd get a "distro" wheel (linux_x86_64-d_ubuntu_14_04) but for > > > SQLAlchemy, you'd get a "libc-generic" wheel > (linux_x86_64-c_glibc_2_19). > > > > > > It's then up to PyPI project owners to build on whatever platforms they > > > wish to support. > > > > > > > I think it's reasonable to not include the libc when the wheel is distro > > specific. I think the barrier to entry on adding new tags is far lower > than > > adding a whole new type of tag. Right now, I think our longest tag is > for OSX > > which is something like macosx_10_10_x86_64 at 19 chars, I don't think > it's > > much worse to have something like linux_glibc_2_19_x86_64 at 23 chars, or > > linux_ubuntu_14_04_x86_64 at 25 chars. I don't think we need the special > c or > > d prefix, we can just treat it as ==, and special case glibc as >= like > we're > > currently special casing the macosx wheels to be >=. > > > > ----------------- > > Donald Stufft > > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From leorochael at gmail.com Tue Sep 8 21:39:26 2015 From: leorochael at gmail.com (Leonardo Rochael Almeida) Date: Tue, 8 Sep 2015 16:39:26 -0300 Subject: [Distutils] Working toward Linux wheel support In-Reply-To: References:

Message-ID: That's nice for singling out some packages (though I only found a , but I had a different use-case in mind, which I guess I didn't fully articulate: I might want binary wheels for some packages, just not coming from PyPI, where I don't necessarily trust whatever was put there. I'm perfectly fine trusting binary wheels coming from my own wheelhouse, for example. So, I'd rather have a: --accept-binary-from=http://mywheelhouse.example.com Which would accept binary from all provided indexes if absent Or perhaps a: --no-binary-from=https://pypi.python.org/simple Regards, Leo On 8 September 2015 at 16:22, Donald Stufft wrote: > On September 8, 2015 at 3:21:26 PM, Leonardo Rochael Almeida ( > leorochael at gmail.com) wrote: > > > > Still, I wouldn't want a random binary wheel suddenly finding its way > into > > my servers, and I would like a way to opt out of it, for "reasons" (ex. I > > might have special build flags, or a special compiler, or maybe I'm still > > waiting for TUF before trusting other peoples binaries on my servers). > > > > ?no-binary packages,that,have,binaries ? > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Sep 9 04:10:55 2015 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 8 Sep 2015 19:10:55 -0700 Subject: [Distutils] Working toward Linux wheel support In-Reply-To: References:

Message-ID: On Mon, Sep 7, 2015 at 9:02 AM, Donald Stufft wrote: > On September 3, 2015 at 1:23:03 PM, Nate Coraor (nate at bx.psu.edu) wrote: >> >>> >> >>> I'll create PRs for this against wheel and pip shortly. I can also work >> >>> on a PEP for the platform tag - I don't think it's going to need to be a >> >>> big one. Are there any preferences as to whether this should be a new PEP >> >>> or an update to 425? >> >>> > > Coming back to this, I'm wondering if we should include the libc > implementation/version in a less generic, but still generic linux wheel. Right > now if you staticly link I think the only platform ABIs you need to worry about > are libc and Python itself. Python itself is handled already but libc is not. > The only thing I've seen so far is "build on an old enough version of glibc > that it handles anything sane", but not all versions of Linux even use glibc at > all. This feels kinda half-baked to me? "linux" is a useful tag because it has a clear meaning: "there exists a linux system somewhere that can run this, but no guarantees about which one, good luck". When building a wheel it's easy to tell whether this tag can be correctly applied. Distro-specific tags are useful because they also have a fairly clear meaning: "here's a specific class of systems that can run this, so long as you install enough packages to fulfill the external dependencies". Again, when building a wheel it's pretty easy to tell whether this tag can be correctly applied. (Of course someone could screw this up, e.g. by building on a system is technically distro X but has some incompatible hand-compiled libraries installed, but 99% of the time we can guess correctly.) If we define a LSB-style base system and give it a tag, like I don't know, the "Python base environment", call it "linux_pybe1_core" or something, that describes what libraries are and aren't available and their ABIs, and provide docs/tooling to help people explicitly create such wheels and check whether they're compatible with their system, then this is also useful -- we have proof that this is sufficient to actually distribute arbitrary software usefully, given that multiple distributors have converged on this strategy. (I've been talking to some people off-list about maybe actually putting together a proposal like this...) To me "linux_glibc_2.18" falls between the cracks though. If this starts being what you get by default when you build a wheel, then people will use it for wheels that are *not* statically linked, and what that tag will mean is "there exists some system that can run this, and that has glibc 2.18 on it, and also some other unspecified stuff, good luck". Which is pretty useless -- we might as well just stick with "linux" in this case. OTOH if it's something that builders have to opt into, then we could document that it's only to be used for wheels that are statically linked except for glibc, and make it mean "*any* system which has glibc 2.18 or later on it can run this". Which would be useful in some cases. But at this point it's basically a version of the "defined base environment" approach, and once you've gone that far you might as well take advantage of the various distributors' experience about what should actually be in that environment -- glibc isn't enough. -n -- Nathaniel J. Smith -- http://vorpus.org From nate at bx.psu.edu Wed Sep 9 17:06:33 2015 From: nate at bx.psu.edu (Nate Coraor) Date: Wed, 9 Sep 2015 11:06:33 -0400 Subject: [Distutils] Working toward Linux wheel support In-Reply-To: References:

Message-ID: On Tue, Sep 8, 2015 at 10:10 PM, Nathaniel Smith wrote: > On Mon, Sep 7, 2015 at 9:02 AM, Donald Stufft wrote: > > On September 3, 2015 at 1:23:03 PM, Nate Coraor (nate at bx.psu.edu) wrote: > >> >>> > >> >>> I'll create PRs for this against wheel and pip shortly. I can also > work > >> >>> on a PEP for the platform tag - I don't think it's going to need to > be a > >> >>> big one. Are there any preferences as to whether this should be a > new PEP > >> >>> or an update to 425? > >> >>> > > > > Coming back to this, I'm wondering if we should include the libc > > implementation/version in a less generic, but still generic linux wheel. > Right > > now if you staticly link I think the only platform ABIs you need to > worry about > > are libc and Python itself. Python itself is handled already but libc is > not. > > The only thing I've seen so far is "build on an old enough version of > glibc > > that it handles anything sane", but not all versions of Linux even use > glibc at > > all. > > This feels kinda half-baked to me? > > "linux" is a useful tag because it has a clear meaning: "there exists > a linux system somewhere that can run this, but no guarantees about > which one, good luck". When building a wheel it's easy to tell whether > this tag can be correctly applied. > I'm not sure how it'd be possible to tell. The same meaning for a generic tag would be true of any wheel built, regardless of whether the wheel has dependencies in addition to libc. > Distro-specific tags are useful because they also have a fairly clear > meaning: "here's a specific class of systems that can run this, so > long as you install enough packages to fulfill the external > dependencies". Again, when building a wheel it's pretty easy to tell > whether this tag can be correctly applied. (Of course someone could > screw this up, e.g. by building on a system is technically distro X > but has some incompatible hand-compiled libraries installed, but 99% > of the time we can guess correctly.) > > If we define a LSB-style base system and give it a tag, like I don't > know, the "Python base environment", call it "linux_pybe1_core" or > something, that describes what libraries are and aren't available and > their ABIs, and provide docs/tooling to help people explicitly create > such wheels and check whether they're compatible with their system, > then this is also useful -- we have proof that this is sufficient to > actually distribute arbitrary software usefully, given that multiple > distributors have converged on this strategy. (I've been talking to > some people off-list about maybe actually putting together a proposal > like this...) > > To me "linux_glibc_2.18" falls between the cracks though. If this > starts being what you get by default when you build a wheel, then > people will use it for wheels that are *not* statically linked, and > what that tag will mean is "there exists some system that can run > this, and that has glibc 2.18 on it, and also some other unspecified > stuff, good luck". Which is pretty useless -- we might as well just > stick with "linux" in this case. OTOH if it's something that builders > have to opt into, then we could document that it's only to be used for > wheels that are statically linked except for glibc, and make it mean > "*any* system which has glibc 2.18 or later on it can run this". Which > would be useful in some cases. This is a tooling issue. If wheel (the package) inspects the built .so files and finds they are not dynamically linked to anything not included with glibc, it can apply the glibc tag. Otherwise, it'd apply the distro tag. There's no possibility for human error here, unless they explicitly override the platform tag. > But at this point it's basically a > version of the "defined base environment" approach, and once you've > gone that far you might as well take advantage of the various > distributors' experience about what should actually be in that > environment -- glibc isn't enough. > While I agree that glibc isn't always enough, defining a base environment that may not be met by the "standard" install of popular distributions makes unprivileged wheel installation much more difficult. It's also not going to work out of the box on older distributions that wouldn't provide whatever standardized mechanism is defined for a list of "base environments currently provided by this system" (unless pip does the work itself at runtime to determine whether a base environment is met). Maybe an important question: how many popular packages with C Extensions have dependencies in addition to glibc? > > -n > > -- > Nathaniel J. Smith -- http://vorpus.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Sep 10 01:49:31 2015 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 9 Sep 2015 16:49:31 -0700 Subject: [Distutils] Working toward Linux wheel support In-Reply-To: References:

Message-ID: On Wed, Sep 9, 2015 at 8:06 AM, Nate Coraor wrote: > On Tue, Sep 8, 2015 at 10:10 PM, Nathaniel Smith wrote: >> >> On Mon, Sep 7, 2015 at 9:02 AM, Donald Stufft wrote: >> > On September 3, 2015 at 1:23:03 PM, Nate Coraor (nate at bx.psu.edu) wrote: >> >> >>> >> >> >>> I'll create PRs for this against wheel and pip shortly. I can also >> >> >>> work >> >> >>> on a PEP for the platform tag - I don't think it's going to need to >> >> >>> be a >> >> >>> big one. Are there any preferences as to whether this should be a >> >> >>> new PEP >> >> >>> or an update to 425? >> >> >>> >> > >> > Coming back to this, I'm wondering if we should include the libc >> > implementation/version in a less generic, but still generic linux wheel. >> > Right >> > now if you staticly link I think the only platform ABIs you need to >> > worry about >> > are libc and Python itself. Python itself is handled already but libc is >> > not. >> > The only thing I've seen so far is "build on an old enough version of >> > glibc >> > that it handles anything sane", but not all versions of Linux even use >> > glibc at >> > all. >> >> This feels kinda half-baked to me? >> >> "linux" is a useful tag because it has a clear meaning: "there exists >> a linux system somewhere that can run this, but no guarantees about >> which one, good luck". When building a wheel it's easy to tell whether >> this tag can be correctly applied. > > > I'm not sure how it'd be possible to tell. The same meaning for a generic > tag would be true of any wheel built, regardless of whether the wheel has > dependencies in addition to libc. Sure... my point is just that "linux" is unambiguous and fills a niche: it unambiguously says "you're on your own", and sometimes that's the best we can hope to say. >> Distro-specific tags are useful because they also have a fairly clear >> meaning: "here's a specific class of systems that can run this, so >> long as you install enough packages to fulfill the external >> dependencies". Again, when building a wheel it's pretty easy to tell >> whether this tag can be correctly applied. (Of course someone could >> screw this up, e.g. by building on a system is technically distro X >> but has some incompatible hand-compiled libraries installed, but 99% >> of the time we can guess correctly.) >> >> If we define a LSB-style base system and give it a tag, like I don't >> know, the "Python base environment", call it "linux_pybe1_core" or >> something, that describes what libraries are and aren't available and >> their ABIs, and provide docs/tooling to help people explicitly create >> such wheels and check whether they're compatible with their system, >> then this is also useful -- we have proof that this is sufficient to >> actually distribute arbitrary software usefully, given that multiple >> distributors have converged on this strategy. (I've been talking to >> some people off-list about maybe actually putting together a proposal >> like this...) >> >> To me "linux_glibc_2.18" falls between the cracks though. If this >> starts being what you get by default when you build a wheel, then >> people will use it for wheels that are *not* statically linked, and >> what that tag will mean is "there exists some system that can run >> this, and that has glibc 2.18 on it, and also some other unspecified >> stuff, good luck". Which is pretty useless -- we might as well just >> stick with "linux" in this case. OTOH if it's something that builders >> have to opt into, then we could document that it's only to be used for >> wheels that are statically linked except for glibc, and make it mean >> "*any* system which has glibc 2.18 or later on it can run this". Which >> would be useful in some cases. > > > This is a tooling issue. If wheel (the package) inspects the built .so files > and finds they are not dynamically linked to anything not included with > glibc, it can apply the glibc tag. Otherwise, it'd apply the distro tag. > There's no possibility for human error here, unless they explicitly override > the platform tag. > >> >> But at this point it's basically a >> version of the "defined base environment" approach, and once you've >> gone that far you might as well take advantage of the various >> distributors' experience about what should actually be in that >> environment -- glibc isn't enough. > > While I agree that glibc isn't always enough, defining a base environment > that may not be met by the "standard" install of popular distributions makes > unprivileged wheel installation much more difficult. Yeah, which is why my suggestion is that we steal the "base environment" definition from the folks like Continuum and Enthought who have already done the work of determining what is in the "standard" install of popular distributions, and have spent years actually distributing packages to unprivileged users :-). > It's also not going to > work out of the box on older distributions that wouldn't provide whatever > standardized mechanism is defined for a list of "base environments currently > provided by this system" (unless pip does the work itself at runtime to > determine whether a base environment is met). Right -- which is basically what pip will have to do to figure out the current glibc version too, right? Trying to guess whether the installed versions of several libraries are really ABI compatible with what we expect is harder than trying to guess whether the installed version of glibc alone is really ABI compatible with what we expect, but in both cases it's basically a heuristic (some distros could have local patches to their glibc that break ABI, who knows) and in both cases it's basically safe to just assume it will work (because if we stick to libraries that other distributors are already depending on then we have years of experience that it pretty much always works). > Maybe an important question: > how many popular packages with C Extensions have dependencies in addition to > glibc? Certainly enough that the major distributors of binary packages on Linux, like Continuum and Enthought, have decided that they need to require more than glibc :-). libstdc++ is an example of one particularly common external dependency. To be clear: if you're talking specifically about the model where we validate that the extensions are statically linked before we enable the glibc tag, then I don't think it will do any harm to have it as an option. It just seems redundant with the more general solution. -n -- Nathaniel J. Smith -- http://vorpus.org From chenchao at inhand.com.cn Tue Sep 8 04:09:10 2015 From: chenchao at inhand.com.cn (chenchao at inhand.com.cn) Date: Tue, 08 Sep 2015 10:09:10 +0800 Subject: [Distutils] Fwd: install In-Reply-To: <55ED69F9.303@inhand.com.cn> References: <55ED69F9.303@inhand.com.cn> Message-ID: <55EE4346.5060507@inhand.com.cn> hi: I cross compiled python2.7.10 and install python on my embedded device. But now, I may install pip on my embedded device. So, I did it as follow: 1>I execute cmd:python -m ensurepip. It ocuurs errors as follow: Traceback (most recent call last): File "/usr/lib/python/lib/python27.zip/runpy.py", line 162, in _run_module_as_main File "/usr/lib/python/lib/python27.zip/runpy.py", line 72, in _run_code File "/var/app/python/libs/ensurepip.zip/ensurepip/__main__.py", line 4, in File "/var/app/python/libs/ensurepip.zip/ensurepip/__init__.py", line 226, in _main File "/var/app/python/libs/ensurepip.zip/ensurepip/__init__.py", line 123, in bootstrap File "/var/app/python/libs/ensurepip.zip/ensurepip/__init__.py", line 45, in _run_pip File "/tmp/tmprHZPx4/pip-6.1.1-py2.py3-none-any.whl/pip/__init__.py", line 15, in File "/tmp/tmprHZPx4/pip-6.1.1-py2.py3-none-any.whl/pip/vcs/subversion.py", line 9, in File "/tmp/tmprHZPx4/pip-6.1.1-py2.py3-none-any.whl/pip/index.py", line 30, in File "/tmp/tmprHZPx4/pip-6.1.1-py2.py3-none-any.whl/pip/_vendor/__init__.py", line 92, in load_module ImportError: No module named 'pip._vendor.html5lib' The message "/tmp/tmprHZPx4/pip-6.1.1-py2.py3-none-any.whl" point out the pip package in that dir, while it is not there. 2>As a result of the above error message, I download the pip-7.1.2 package and install it on my embedded device. It ocuurs errors as follow: # python setup.py install Traceback (most recent call last): File "setup.py", line 6, in from setuptools import setup, find_packages ImportError: No module named setuptools # 3> I I download the setuptools package and execute cmd: python setup.py install.It ocuurs errors as follow: # python setup.py install Traceback (most recent call last): File "setup.py", line 6, in from setuptools import setup, find_packages File "/var/app/python/libs/setuptools.zip/setuptools/__init__.py", line 12, in File "/var/app/python/libs/setuptools.zip/setuptools/extension.py", line 8, in File "/var/app/python/libs/setuptools.zip/setuptools/dist.py", line 19, in ImportError: No module named pkg_resources So, where can i download the pkg_resources package. Is there somebody installed pip on embedded device? can you tell me how to do that? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dw+python-ideas at hmmz.org Thu Sep 10 01:01:30 2015 From: dw+python-ideas at hmmz.org (David Wilson) Date: Wed, 9 Sep 2015 23:01:30 +0000 Subject: [Distutils] [Python-ideas] PyPI search still broken In-Reply-To: References: <204080FC-D5B0-44A9-9D9D-582B1491B413@yahoo.com> <20150904172710.GO19373@ando.pearwood.info> <87lhcl2viv.fsf@uwakimon.sk.tsukuba.ac.jp>

<85h9n482sa.fsf@benfinney.id.au>

Message-ID: <20150909230130.GA14415@k3> Hi there, My 2.5 year old offer to retrofit the old codebase with a new search system still stands[1]. :) There is no reason for this to be a complex affair, the prototype built back then took only a few hours to complete. No doubt the long term answer is probably "Warehouse fixes this", but Warehouse seems no nearer a reality than it did in 2013. David [1] https://groups.google.com/forum/#!search/%22david$20wilson%22$20search$20pypi/pypa-dev/ZjUNkczsKos/2et8926YOQYJ On Thu, Sep 10, 2015 at 12:35:04AM +0200, Giovanni Cannata wrote: > Hi, sorry to bother you again, but the search problem on PyPI is still present > after different weeks and it's very annoying. I've just released a new version > of my ldap3 project and it doesn't show up when searching with its name. For > mine (and I suppose for other emerging project, especially related to Python 3) > it's vital to be easily found by other developers that use pip and PyPI as THE > only repository for python packages and using the number of download as a > ranking of popularity of a project. > > If search can't be fixed there should be at least a warning on the PyPI > homepage to let users know that search is broken and that using Google for > searching could help to find more packages. > > Bye, > Giovanni > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From holger at merlinux.eu Thu Sep 10 13:11:57 2015 From: holger at merlinux.eu (holger krekel) Date: Thu, 10 Sep 2015 11:11:57 +0000 Subject: [Distutils] devpi-server-2.3.0: changed pypi caching, semantic versioning Message-ID: <20150910111157.GM26059@merlinux.eu> devpi-server-2.3: more resilient pypi caching, semantic versioning ============================================================================ The devpi-server-2.3 release brings two important changes: - We don't use the XMLRPC "changelog" interface of pypi.python.org anymore. Instead devpi-server now re-checks with pypi every 30 minutes and will serve cached contents in between. If pypi is not reachable and we still have cached contents we continue serving the cached content so that we can survive pypi outages. - we switched to semantic versioning so that only major version number increases signal the need for an export/import cycle. If you have a devpi-server-2.2.X installation you are not required to export/import. However, there has been a regression with the execnet-1.4.0 release which was fixed with the now current execnet-1.4.1 release. If you have freshly setup devpi-server and used execnet-1.4.0 at that time you will need to do an export with execnet-1.4.0 and then import with execnet-1.4.1 installed. Note also that we released new micro releases of devpi-client and devpi-web which are pure bugfixing releases. For many more changes and fixes, please see the respective CHANGELOG of the respective packages. For docs and quickstart tutorials see http://doc.devpi.net many thanks to Florian Schulze who any of the new features. And special thanks go to the two still unnamed companies who funded major parts of the above work. have fun, holger krekel, merlinux GmbH 2.3.0 (2015-09-10) ------------------ - switched to semantic versioning. Only major revisions will ever require an export/import cycle. - fix issue260: Log identical upload message on level "info" - Log upload trigger message on level "warn" - The PyPI changelog isn't watched for changes anymore. Instead we cache release data for 30 minutes, this can be adjusted with the ``--mirror-cache-expiry`` option. - fix issue251: Require and validate the "X-DEVPI-SERIAL" from master in replica thread - fix issue258: fix FileReplicationError representation for proper logging - fix issue256: if a project removes all releases from pypi or the project is deleted on pypi, we get a 404 back. In that case we now return an empty list of releases instead of returning an UpstreamError. - Change nginx template to serve HEAD in addition to GET requests of files directly instead of proxying to devpi-server - make keyfs cache size configurable via "--keyfs-cache-size" option and increase the default size to improve performance for installations with many writes From contact at ionelmc.ro Thu Sep 10 14:07:14 2015 From: contact at ionelmc.ro (=?UTF-8?Q?Ionel_Cristian_M=C4=83rie=C8=99?=) Date: Thu, 10 Sep 2015 15:07:14 +0300 Subject: [Distutils] [Python-ideas] PyPI search still broken In-Reply-To: <20150909230130.GA14415@k3> References: <204080FC-D5B0-44A9-9D9D-582B1491B413@yahoo.com> <20150904172710.GO19373@ando.pearwood.info> <87lhcl2viv.fsf@uwakimon.sk.tsukuba.ac.jp>

<85h9n482sa.fsf@benfinney.id.au>

<20150909230130.GA14415@k3> Message-ID: Wouldn't it be better if you'd just build an external search service? Getting a list of packages and descriptions should be possible no? (just asking, not 100% sure) I doubt the maintainers are just going to come out and say "ok, this guy has waited long enough, lets take his contribution in". If they didn't care about the search 2.5 years ago why would they care now. Sorry for being snide here but my impression is that Warehouse could had been shipped a while ago instead of getting rewritten ? ? s ?everal times.? I'm not saying that's bad, it's just that there's a mismatch in goals here. Thanks, -- Ionel Cristian M?rie? On Thu, Sep 10, 2015 at 2:01 AM, David Wilson wrote: > Hi there, > > My 2.5 year old offer to retrofit the old codebase with a new search > system still stands[1]. :) There is no reason for this to be a complex > affair, the prototype built back then took only a few hours to complete. > > No doubt the long term answer is probably "Warehouse fixes this", but > Warehouse seems no nearer a reality than it did in 2013. > > > David > > [1] > https://groups.google.com/forum/#!search/%22david$20wilson%22$20search$20pypi/pypa-dev/ZjUNkczsKos/2et8926YOQYJ > > On Thu, Sep 10, 2015 at 12:35:04AM +0200, Giovanni Cannata wrote: > > Hi, sorry to bother you again, but the search problem on PyPI is still > present > > after different weeks and it's very annoying. I've just released a new > version > > of my ldap3 project and it doesn't show up when searching with its name. > For > > mine (and I suppose for other emerging project, especially related to > Python 3) > > it's vital to be easily found by other developers that use pip and PyPI > as THE > > only repository for python packages and using the number of download as a > > ranking of popularity of a project. > > > > If search can't be fixed there should be at least a warning on the PyPI > > homepage to let users know that search is broken and that using Google > for > > searching could help to find more packages. > > > > Bye, > > Giovanni > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dw+python-ideas at hmmz.org Thu Sep 10 14:47:46 2015 From: dw+python-ideas at hmmz.org (David Wilson) Date: Thu, 10 Sep 2015 12:47:46 +0000 Subject: [Distutils] [Python-ideas] PyPI search still broken In-Reply-To: References: <87lhcl2viv.fsf@uwakimon.sk.tsukuba.ac.jp>

<85h9n482sa.fsf@benfinney.id.au>

<20150909230130.GA14415@k3> Message-ID: <20150910124746.GA18845@k3> On Thu, Sep 10, 2015 at 03:07:14PM +0300, Ionel Cristian M?rie? wrote: > Wouldn't it be better if you'd just build an external search service? > Getting a list of packages and descriptions should be possible no? > (just asking, not 100% sure) That would be the idea. In fact preferably not build a service at all, just pay someone $50/mo for hosted ElasticSearch, rip out the guts of the old thing and write a small sync cron job similar to the one existing in the Bitbucket repo I linked. David From donald at stufft.io Thu Sep 10 15:31:13 2015 From: donald at stufft.io (Donald Stufft) Date: Thu, 10 Sep 2015 09:31:13 -0400 Subject: [Distutils] [Python-ideas] PyPI search still broken In-Reply-To: <20150910124746.GA18845@k3> References: <87lhcl2viv.fsf@uwakimon.sk.tsukuba.ac.jp>

<85h9n482sa.fsf@benfinney.id.au>

<20150909230130.GA14415@k3> <20150910124746.GA18845@k3> Message-ID: On September 10, 2015 at 8:48:05 AM, David Wilson (dw+python-ideas at hmmz.org) wrote: > On Thu, Sep 10, 2015 at 03:07:14PM +0300, Ionel Cristian M?rie? wrote: > > > Wouldn't it be better if you'd just build an external search service? > > Getting a list of packages and descriptions should be possible no? > > (just asking, not 100% sure) > > That would be the idea. In fact preferably not build a service at all, > just pay someone $50/mo for hosted ElasticSearch, rip out the guts of > the old thing and write a small sync cron job similar to the one > existing in the Bitbucket repo I linked. > > The old PostgreSQL based system has been gone for awhile, and we already have ElasticSearch with a small cron job that runs every 3 hours to index the data. When we moved the database to Heroku this cronjob started taking 6+ hours to complete, because we were fetching data in too small of chunks which didn't actually hurt when the script and the database were running close to each other. That got "fixed" a day or two ago by increasing the size of the chunks we pulled from 1000 to 10000 and by switching to a SERIALIZABLE READ ONLY DEFERRABLE transaction so that we only needed to hold open a lock right at the very beginning which has the job finishing in 40 minutes now. I suspect further enhancements to the indexing speed will require? locating the script in EC2 to get it closer to the PostgreSQL instance. Given that these problems seem to be *new* since the move of the database to Heroku, I don't think the shape of our data in Elasticsearch nor the actual query we're using which hasn't changed should be at fault, so I've been trying to figure out what else we might have changed in the transition that would have caused it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From benjaminrk at gmail.com Thu Sep 10 15:11:17 2015 From: benjaminrk at gmail.com (MinRK) Date: Thu, 10 Sep 2015 15:11:17 +0200 Subject: [Distutils] platform_python_implementation not implemented. Message-ID: Hello, I?m working on specifying dependencies for a project (IPython) that are dependent on the Python implementation - we want to depend on a package on CPython, but not on PyPy. I see from PEPs 345 and 426 and 496 that this should be available as platform_python_version environment marker, but when I try to use this in setup.py, it fails, claiming this is an invalid marker (setuptools 18.3.1). I discovered that pkg_resources has its own implementation of environment markers , which isn?t consistent with any PEPs describing them. pip uses markerlib, which does seem to implement PEP 345 correctly. The relevant difference in this case is that pkg_resources misspells platform_python_implementation as python_implementation, but it is not the only one. Due to the inconsistent implementations, I don?t think there?s a way to use this environment marker anywhere. It seems like the whole concept of environment markers is experimental, and it would be premature to adopt them for any packages in production. Is this the case? I found that when I run pip install ., the pkg_resources version is used, and it will balk at the correct platform_python_version as invalid. However, when I build a wheel and try to install it with pip install ipython-...whl, the pip version is used, and it balks at pkg_resources incorrect python_implementation. This conflict makes it impossible to use the marker, as far as I can tell. Has anyone been able to use the python implementation environment marker? I have a PR to setuptools to fix what seems to be some inconsistency with the PEPs while preserving the misspelling for backward-compatibility, but it?s not clear which metadata/env marker PEPs setuptools and/or pip are meant to support at this point. It?s also unclear why pkg_resources has its own implementation, instead of all participants using the shared _markerlib, which would at least avoid inconsistency. Thanks, -MinRK ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dw+python-ideas at hmmz.org Thu Sep 10 15:51:49 2015 From: dw+python-ideas at hmmz.org (David Wilson) Date: Thu, 10 Sep 2015 13:51:49 +0000 Subject: [Distutils] [Python-ideas] PyPI search still broken In-Reply-To: References: <85h9n482sa.fsf@benfinney.id.au>

<20150909230130.GA14415@k3> <20150910124746.GA18845@k3> Message-ID: <20150910135149.GA28704@k3> On Thu, Sep 10, 2015 at 09:31:13AM -0400, Donald Stufft wrote: > The old PostgreSQL based system has been gone for awhile, and we > already have ElasticSearch with a small cron job that runs every 3 > hours to index the data. That's awesome news. :) David From dpoirier at caktusgroup.com Thu Sep 10 15:57:51 2015 From: dpoirier at caktusgroup.com (Dan Poirier) Date: Thu, 10 Sep 2015 09:57:51 -0400 Subject: [Distutils] [Python-ideas] PyPI search still broken In-Reply-To: References: <87lhcl2viv.fsf@uwakimon.sk.tsukuba.ac.jp>