[Distutils] Re: Environment markers for GPU/CUDA availibility

4 Sep 2018

      On Tue, 4 Sep 2018 at 20:30, Nathaniel Smith  wrote:
...
On Mon, Sep 3, 2018 at 4:51 PM, Nick Coghlan  wrote:
...
On Mon., 3 Sep. 2018, 5:48 am Ronald Oussoren, 
wrote:
...
What’s the problem with including GPU and non-GPU variants of code in a
binary wheel other than the size of the wheel? I tend to prefer binaries
that work “everywhere", even if that requires some more work in building
binaries (such as including multiple variants of extensions to have
optimised code for different CPU variants, such as SSE and non-SSE variants
in the past).
As far as I'm aware, binary artifact size *is* the problem. It's just that
once you're  automatically building and pushing an artifact (or an image
containing that artifact) to thousands or tens of thousands of managed
systems, the wasted bandwidth from pushing redundant implementations of the
same functionality becomes more of a concern than the convenience of being
able to use the same artifact across multiple platforms.
None of the links that Dustin gave at the top of the thread are about
managed systems though.
When you're only managing a few systems, or only saving a few MB per
download, "install both and pick at runtime" is an entirely viable
option.

However, since tensorflow is the example, neither of those cases is true:

1. It's a Google project, so they have tens of thousands of instances
to worry about (as do other cloud providers)
2. The size difference is in the tens or even hundreds of megabytes

I didn't actually realise the GPU tensorflow package was over 200 MB,
though - that's large enough to be noticeably slow to extract and
install even from a local zipfile, let alone if you're needing to
download it first.

Cheers,
Nick.

P.S. If anyone were to be worrying about this problem in the context
of PyPI specifically, the most likely candidates would actually be
Fastly, and then to a lesser degree, AWS, as they're handling the bulk
of any data transfer costs associated with folks downloading packages
that are larger than they really need to be :)

-- 
Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia