From andrea at andreabedini.com Mon Jun 1 04:11:20 2015 From: andrea at andreabedini.com (Andrea Bedini) Date: Mon, 1 Jun 2015 12:11:20 +1000 Subject: [Distutils] pip can't find header file for extension module, but `python setup.py install` works fine In-Reply-To: References: Message-ID: <093C3080-F7F7-4D46-80D3-976EA9E46E02@andreabedini.com> Hi AJ, > On 1 Jun 2015, at 7:07 am, AJ Friend wrote: > I'm trying to write a new `setup.py` file for an extension module to > wrap a C library (https://github.com/cvxgrp/scs). I have been looking for the best way to do this too and I am keen to hear from distutils-sig. I can show you my currently favourite way, although I am not sure it?s applicable to your case. class BuildExtensions(build_ext): def run(self): # At this point we can be sure pip has already installed numpy numpy_incl = pkg_resources.resource_filename('numpy', 'core/include') for ext in self.extensions: if (hasattr(ext, 'include_dirs') and numpy_incl not in ext.include_dirs): ext.include_dirs.append(numpy_incl) build_ext.run(self) and then of course pass requires=[?numpy?] to setup. This avoids running any of numpy?s code. It should work because as of version 6.1.0 pip installs project dependencies in the right (topological) order. This should be enough for your version but the original repo was importing numpy also to access other informations. from numpy.distutils.system_info import get_info, BlasNotFoundError and I am not sure how to access that without importing numpy. > Any idea why that would be happening? Could it have anything to do > with the relative path I'm giving for the include directories? This is impossible to say without a build log. Can you upload one somewhere (gist/pastebin/etc) ? Andrea -- Andrea Bedini @andreabedini, http://www.andreabedini.com use https://keybase.io/andreabedini to send me private messages From erik.m.bray at gmail.com Mon Jun 1 16:51:58 2015 From: erik.m.bray at gmail.com (Erik Bray) Date: Mon, 1 Jun 2015 10:51:58 -0400 Subject: [Distutils] pip can't find header file for extension module, but `python setup.py install` works fine In-Reply-To: References: Message-ID: On Sun, May 31, 2015 at 5:07 PM, AJ Friend wrote: > Hi, > > I'm trying to write a new `setup.py` file for an extension module to > wrap a C library (https://github.com/cvxgrp/scs). > > The current `setup.py` file imports numpy. I'm trying to delay that > import statement until setuptools has a chance to install numpy if > it's not already installed. I'm trying to do that with this bit of > code: > > from setuptools.command.build_ext import build_ext as _build_ext > class build_ext(_build_ext): > def finalize_options(self): > _build_ext.finalize_options(self) > # Prevent numpy from thinking it is still in its setup process: > __builtins__.__NUMPY_SETUP__ = False > import numpy > self.include_dirs += ext['include_dirs'] + [numpy.get_include()] > > Running `python setup.py install` seems to work fine on my OSX > machine, but when I run `pip install .` in the directory with > `setup.py`, I get a clang error that it can't find one of the header > files. > > Any idea why that would be happening? Could it have anything to do > with the relative path I'm giving for the include directories? > > Also, I had trouble finding good documentation on subclassing > build_ext. Does anyone know if setting self.include_dirs overwrites or > appends to the include_dirs attribute of an Extension object defined > later in setup.py? > > For the curious, my current attempt at setup.py is > athttps://github.com/ajfriend/scs/blob/setup2/python/setup.py. The > original can be found in the same directory. > > More generally, since I'm new to python packaging, I'm not sure how > well or correctly I've written my `setup.py` file. Any feedback on > doing things correctly would be appreciated. Hi AJ, For a lot of things in Python packaging there is not, sadly, One Right Way to Do It. Your setup.py looks okay though. You may want to have a look at the get_numpy_include_path utility here: https://github.com/astropy/astropy-helpers/blob/7ee7e543641759ed1ee2b691bba1378cec76a001/astropy_helpers/utils.py#L66 It's similar to what you're already doing, but maybe a little more 'robust'. In particular, I think the reload of the numpy module may be important. Erik From org.python.distutils-sig at pooryorick.com Tue Jun 2 03:04:47 2015 From: org.python.distutils-sig at pooryorick.com (Poor Yorick) Date: Tue, 2 Jun 2015 01:04:47 +0000 Subject: [Distutils] distutils package includes, package system includes, and environment-defined system includes Message-ID: <20150602010446.GG25939@web106.webfaction.com> I maintain a software collection that installs to an alternate prefix, and some of the longstanding issues I've seen with Python packages is that distutils doesn't seem to have any mechanism to allow a package build script to distinguish between project include directories and system include directories, and therefore include_dirs gets used for both purposes. The main issue is that project include directories should preceed anything specified in CPPFLAGS, whereas system include directories should follow CPPFLAGS. I've had a look at distutils2 and the situation is pretty much the same there. I finally decided to spend some time exploring this issue, and ended up creating a fairly extensive modification of the distutils that was shipped with Python-2.7.9, and also of the numpy and scipy packages. They take the approach of having cc_args precede the CPPFLAGS. This leaves extra_compile_args as a possible mechanism for system includes. Links to my patches are below. So far I like the effect they've had. Adding parsed environment variables to "extensions" is a little beyond its stated scope, but it has the benefit of unifying the treatment of command line options and of giving the user the opportunity to bind options tightly to the compiler command by adding options there, or to leave them, for example, in CFLAGS, where they can potentially be overriden by the package distribution. I also like the fact that the modifications make "compiler_exe" and "compiler_so" disappear, and leave "compiler", "compiler_flags_so", and "compiler_flags_exe" instead. That seems like a a cleaner model of the commands involved. I'm considering adding a "sysincludedirs" argument to Extension(). Using the disutils packaged with Python-2.7.9, numpy wasn't doing the right thing, and compiler flags weren't making it into the Fortran command lines. with these modifications and the numpy modifications, things are working better. I would appreciate any feedback anyone interested might have after glancing over the patches. http://dpaste.com/215AF8J http://dpaste.com/0K94JJ2 -- Yorick From robertc at robertcollins.net Tue Jun 2 09:41:51 2015 From: robertc at robertcollins.net (Robert Collins) Date: Tue, 2 Jun 2015 19:41:51 +1200 Subject: [Distutils] pip can't find header file for extension module, but `python setup.py install` works fine In-Reply-To: References: Message-ID: On 2 June 2015 at 02:51, Erik Bray wrote: > On Sun, May 31, 2015 at 5:07 PM, AJ Friend wrote: >> Hi, >> >> I'm trying to write a new `setup.py` file for an extension module to >> wrap a C library (https://github.com/cvxgrp/scs). >> >> The current `setup.py` file imports numpy. I'm trying to delay that >> import statement until setuptools has a chance to install numpy if >> it's not already installed. I'm trying to do that with this bit of >> code: >> >> from setuptools.command.build_ext import build_ext as _build_ext >> class build_ext(_build_ext): >> def finalize_options(self): >> _build_ext.finalize_options(self) >> # Prevent numpy from thinking it is still in its setup process: >> __builtins__.__NUMPY_SETUP__ = False >> import numpy >> self.include_dirs += ext['include_dirs'] + [numpy.get_include()] >> >> Running `python setup.py install` seems to work fine on my OSX >> machine, but when I run `pip install .` in the directory with >> `setup.py`, I get a clang error that it can't find one of the header >> files. >> >> Any idea why that would be happening? Could it have anything to do >> with the relative path I'm giving for the include directories? >> >> Also, I had trouble finding good documentation on subclassing >> build_ext. Does anyone know if setting self.include_dirs overwrites or >> appends to the include_dirs attribute of an Extension object defined >> later in setup.py? >> >> For the curious, my current attempt at setup.py is >> athttps://github.com/ajfriend/scs/blob/setup2/python/setup.py. The >> original can be found in the same directory. >> >> More generally, since I'm new to python packaging, I'm not sure how >> well or correctly I've written my `setup.py` file. Any feedback on >> doing things correctly would be appreciated. > > Hi AJ, > > For a lot of things in Python packaging there is not, sadly, One Right > Way to Do It. Your setup.py looks okay though. > > You may want to have a look at the get_numpy_include_path utility here: > https://github.com/astropy/astropy-helpers/blob/7ee7e543641759ed1ee2b691bba1378cec76a001/astropy_helpers/utils.py#L66 > > It's similar to what you're already doing, but maybe a little more > 'robust'. In particular, I think the reload of the numpy module may > be important. It is, because what numpy is doing is importing from its own tree before setup has completed - which is in general unsafe. And numpy may be in process depending on exactly what setuptools setup_requires easy-install code has done, which can lead to the symptoms you see. This should be better once we get pip interpreting setup_requires for you, but a perhaps shorter term fix would be to factor out the distutils support glue from numpy (which seems pretty self contained) into a out of package file - either adjacent to setup.py, or, if numpy is willing to make the dependency on setuptools a hard dep (which I recommend), then as a separate package which can be setup_require'd. That would avoid the bootstrapping complexity, resulting in numpy not being imported until it has actually been installed, and things should be rosy for folk downstream of numpy from that point on. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From erik.m.bray at gmail.com Tue Jun 2 18:12:48 2015 From: erik.m.bray at gmail.com (Erik Bray) Date: Tue, 2 Jun 2015 12:12:48 -0400 Subject: [Distutils] pip can't find header file for extension module, but `python setup.py install` works fine In-Reply-To: References: Message-ID: On Tue, Jun 2, 2015 at 3:41 AM, Robert Collins wrote: > On 2 June 2015 at 02:51, Erik Bray wrote: >> On Sun, May 31, 2015 at 5:07 PM, AJ Friend wrote: >>> Hi, >>> >>> I'm trying to write a new `setup.py` file for an extension module to >>> wrap a C library (https://github.com/cvxgrp/scs). >>> >>> The current `setup.py` file imports numpy. I'm trying to delay that >>> import statement until setuptools has a chance to install numpy if >>> it's not already installed. I'm trying to do that with this bit of >>> code: >>> >>> from setuptools.command.build_ext import build_ext as _build_ext >>> class build_ext(_build_ext): >>> def finalize_options(self): >>> _build_ext.finalize_options(self) >>> # Prevent numpy from thinking it is still in its setup process: >>> __builtins__.__NUMPY_SETUP__ = False >>> import numpy >>> self.include_dirs += ext['include_dirs'] + [numpy.get_include()] >>> >>> Running `python setup.py install` seems to work fine on my OSX >>> machine, but when I run `pip install .` in the directory with >>> `setup.py`, I get a clang error that it can't find one of the header >>> files. >>> >>> Any idea why that would be happening? Could it have anything to do >>> with the relative path I'm giving for the include directories? >>> >>> Also, I had trouble finding good documentation on subclassing >>> build_ext. Does anyone know if setting self.include_dirs overwrites or >>> appends to the include_dirs attribute of an Extension object defined >>> later in setup.py? >>> >>> For the curious, my current attempt at setup.py is >>> athttps://github.com/ajfriend/scs/blob/setup2/python/setup.py. The >>> original can be found in the same directory. >>> >>> More generally, since I'm new to python packaging, I'm not sure how >>> well or correctly I've written my `setup.py` file. Any feedback on >>> doing things correctly would be appreciated. >> >> Hi AJ, >> >> For a lot of things in Python packaging there is not, sadly, One Right >> Way to Do It. Your setup.py looks okay though. >> >> You may want to have a look at the get_numpy_include_path utility here: >> https://github.com/astropy/astropy-helpers/blob/7ee7e543641759ed1ee2b691bba1378cec76a001/astropy_helpers/utils.py#L66 >> >> It's similar to what you're already doing, but maybe a little more >> 'robust'. In particular, I think the reload of the numpy module may >> be important. > > It is, because what numpy is doing is importing from its own tree > before setup has completed - which is in general unsafe. > > And numpy may be in process depending on exactly what setuptools > setup_requires easy-install code has done, which can lead to the > symptoms you see. This should be better once we get pip interpreting > setup_requires for you, but a perhaps shorter term fix would be to > factor out the distutils support glue from numpy (which seems pretty > self contained) into a out of package file - either adjacent to > setup.py, or, if numpy is willing to make the dependency on setuptools > a hard dep (which I recommend), then as a separate package which can > be setup_require'd. > > That would avoid the bootstrapping complexity, resulting in numpy not > being imported until it has actually been installed, and things should > be rosy for folk downstream of numpy from that point on. That might not be a bad idea. This is more or less what we did for Astropy with astropy-helpers (this was done also so that other software distributions in the Astropy affiliated package ecosystem could take advantage of it). I think maybe splitting out much of the stuff in numpy.distutils into its own package (though unfortunately it would need a new name since I don't think we could make numpy into a namespace package) might help. Though I'm not sure how that would work for numpy.get_includes(). Would it just figure out where the include files *would* be installed to if one were installing Numpy into the currently active environment? Erik From tseaver at palladion.com Wed Jun 3 19:53:56 2015 From: tseaver at palladion.com (Tres Seaver) Date: Wed, 03 Jun 2015 13:53:56 -0400 Subject: [Distutils] 'virtualenv -p jython' leaves 'pip' non-executable Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Trying to get 'tox -e jython' to work: - ---------------------------- %< ---------------------------- $ file `which jython` /home/tseaver/bin/jython: symbolic link to `/opt/jython-2.7.0/bin/jython' $ /opt/Python-2.7.9/bin/pip --version pip 7.0.3 from /opt/Python-2.7.9/lib/python2.7/site-packages/pip-7.0.3-py2.7.egg (python 2.7) $ /opt/Python-2.7.9/bin/virtualenv --version 13.0.3 $ /opt/Python-2.7.9/bin/tox --version 2.0.2 imported from /opt/Python-2.7.9/lib/python2.7/site-packages/tox-2.0.2-py2.7.egg/tox/__init__.pyc $ git clone git at github.com:zopefoundation/zope.event Cloning into 'zope.event'... remote: Counting objects: 531, done. remote: Total 531 (delta 0), reused 0 (delta 0), pack-reused 531 Receiving objects: 100% (531/531), 84.80 KiB, done. Resolving deltas: 100% (288/288), done. $ /opt/Python-2.7.9/bin/tox -e jython GLOB sdist-make: /tmp/zope.event/setup.py jython create: /tmp/zope.event/.tox/jython jython inst: /tmp/zope.event/.tox/dist/zope.event-4.0.4.dev0.zip ERROR: invocation failed (errno 13), args: [local('/tmp/zope.event/.tox/jython/bin/pip'), 'install', '/tmp/zope.event/.tox/dist/zope.event-4.0.4.dev0.zip'], cwd: /tmp/zope.event Traceback (most recent call last): File "/opt/Python-2.7.9/bin/tox", line 9, in load_entry_point('tox==2.0.2', 'console_scripts', 'tox')() ... File "/opt/Python-2.7.9/lib/python2.7/subprocess.py", line 1335, in _execute_child raise child_exception OSError: [Errno 13] Permission denied $ ls -laF .tox/jython/bin total 64 drwxrwxr-x 2 tseaver tseaver 4096 Jun 3 13:40 ./ drwxrwxr-x 6 tseaver tseaver 4096 Jun 3 13:40 ../ - -rw-rw-r-- 1 tseaver tseaver 2208 Jun 3 13:40 activate - -rw-rw-r-- 1 tseaver tseaver 1264 Jun 3 13:40 activate.csh - -rw-rw-r-- 1 tseaver tseaver 2477 Jun 3 13:40 activate.fish - -rw-rw-r-- 1 tseaver tseaver 1137 Jun 3 13:40 activate_this.py - -rw-rw-r-- 1 tseaver tseaver 255 Jun 3 13:40 easy_install - -rw-rw-r-- 1 tseaver tseaver 255 Jun 3 13:40 easy_install-2.7 - -rwxrwxr-x 1 tseaver tseaver 15055 Jun 3 13:40 jython* - -rw-rw-r-- 1 tseaver tseaver 227 Jun 3 13:40 pip - -rw-rw-r-- 1 tseaver tseaver 227 Jun 3 13:40 pip2 - -rw-rw-r-- 1 tseaver tseaver 227 Jun 3 13:40 pip2.7 lrwxrwxrwx 1 tseaver tseaver 6 Jun 3 13:40 python -> jython* lrwxrwxrwx 1 tseaver tseaver 6 Jun 3 13:40 python2 -> jython* lrwxrwxrwx 1 tseaver tseaver 6 Jun 3 13:40 python2.7 -> jython* - -rw-rw-r-- 1 tseaver tseaver 234 Jun 3 13:40 wheel - ---------------------------- %< ---------------------------- If I make the 'pip' script executable, then 'tox -e jython' runs: - ---------------------------- %< ---------------------------- $ chmod a+x .tox/jython/bin/pip $ tox -e jython GLOB sdist-make: /tmp/zope.event/setup.py jython inst-nodeps: /tmp/zope.event/.tox/dist/zope.event-4.0.4.dev0.zip jython installed: -f file:///home/tseaver/.pip/wheels,wheel==0.24.0,zope.event==4.0.4.dev0 jython runtests: PYTHONHASHSEED='441626771' jython runtests: commands[0] | python setup.py -q test -q warning: no previously-included files matching '*.dll' found anywhere in distribution warning: no previously-included files matching '*.pyc' found anywhere in distribution warning: no previously-included files matching '*.pyo' found anywhere in distribution warning: no previously-included files matching '*.so' found anywhere in distribution warning: no previously-included files matching '*.class' found anywhere in distribution .. - ---------------------------------------------------------------------- Ran 2 tests in 0.004s OK ___________________________________ summary ____________________________________ jython: commands succeeded congratulations :) - ---------------------------- %< ---------------------------- So, leaving tox out of the picture: - ---------------------------- %< ---------------------------- $ /opt/Python-2.7.9/bin/virtualenv -p jython foobar Running virtualenv with interpreter /home/tseaver/bin/jython Cannot find file /opt/jython-2.7.0/Include (bad symlink) New jython executable in foobar/bin/jython Installing setuptools, pip, wheel...done. $ ll foobar/bin/ total 64 drwxrwxr-x 2 tseaver tseaver 4096 Jun 3 13:46 ./ drwxrwxr-x 5 tseaver tseaver 4096 Jun 3 13:45 ../ - -rw-rw-r-- 1 tseaver tseaver 2192 Jun 3 13:46 activate - -rw-rw-r-- 1 tseaver tseaver 1248 Jun 3 13:46 activate.csh - -rw-rw-r-- 1 tseaver tseaver 2461 Jun 3 13:46 activate.fish - -rw-rw-r-- 1 tseaver tseaver 1137 Jun 3 13:46 activate_this.py - -rw-rw-r-- 1 tseaver tseaver 239 Jun 3 13:46 easy_install - -rw-rw-r-- 1 tseaver tseaver 239 Jun 3 13:46 easy_install-2.7 - -rwxrwxr-x 1 tseaver tseaver 15055 Jun 3 13:45 jython* - -rw-rw-r-- 1 tseaver tseaver 211 Jun 3 13:46 pip - -rw-rw-r-- 1 tseaver tseaver 211 Jun 3 13:46 pip2 - -rw-rw-r-- 1 tseaver tseaver 211 Jun 3 13:46 pip2.7 lrwxrwxrwx 1 tseaver tseaver 6 Jun 3 13:45 python -> jython* lrwxrwxrwx 1 tseaver tseaver 6 Jun 3 13:45 python2 -> jython* lrwxrwxrwx 1 tseaver tseaver 6 Jun 3 13:45 python2.7 -> jython* - -rw-rw-r-- 1 tseaver tseaver 218 Jun 3 13:46 wheel - ---------------------------- %< ---------------------------- None of the scripts are being marked as executable. I don't know whether this is a virtualenv issue, a jython issue, or a setuptools issue. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iQIcBAEBAgAGBQJVbz80AAoJEPKpaDSJE9HYg2EP/2sWlshPGokrrhOZYV/8YgcM IFNa2JlwKNwPefKtQ7VThrIqYHCTo8UuLC+6mv1JFvXIT50dTubOFeLP2ddR9X1f Ma5kqwlCs+lf1VhqWbA5BIkg+p3GH+l/CImudJOijxF2Lvlx1sHvIOTZwgU1qYvG cpMAOW5D9rT34oU7SEf6mIGro8cGVa3nEMsp8optU7iOZTzvgQG1nyYknfBJquca jwjwBOgrxaZkIaYrCyFWspR8sdPaWs+9S4qR7RsXkCWYHbfjtNKObJB4CPw4VdIV S0M16pz+znrY3nBggrCFzLAyJQje2E5mwNF1BDMKCrWTbUbdLmpbWMdV0//A68kP yricqNoH5Juw5PVCl5AiP49mIAvBUWipyD57kUVNP4ndlU23DMiC7/ClqS66HSw/ HCiThu50wVwZ4SYX/YOGmPorQKtmntLbLhGx2Oj1f74YoqVtcrTUbqeFG0MIW2W5 EU8IHy3k9n8hVacP9bm0zipUAX5/AJe+vS8eA+14B9pbU690hLK4pQZ8EYM2Dn87 mT7ii9IKgXUElybMWqR5v1w3kwbj/lRZbFqJ59x2cE0Kar1z5P95Eij6GgRP5sBw 6EdtK/35HVOk5K51Wkd7AWfVawSYRV9NoPxFFNcj3l5MqyQI0ewt85KpuYdeHYv3 w/9dWcP6PNXmDxA2VXA6 =eZat -----END PGP SIGNATURE----- From tseaver at palladion.com Thu Jun 4 14:11:58 2015 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 04 Jun 2015 08:11:58 -0400 Subject: [Distutils] 'virtualenv -p jython' leaves 'pip' non-executable In-Reply-To: <55700192.3010300@yahoo.com> References: <55700192.3010300@yahoo.com> Message-ID: <5570408E.8070903@palladion.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/04/2015 03:43 AM, Suresh V. wrote: > Could it be your umask? Just wondering... > Nope. This issue only affects the 'jython' environment -- all other tox environments run fine. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iQIcBAEBAgAGBQJVcECOAAoJEPKpaDSJE9HY73cP/ip5pY1NxhdOo0jAHACQxGGN U4bhzcNm0Qy+WGHQfCfNnIVAVMNgAdFV1nfI1Uu6x4I4lS6Ljm5Q1lXn4k4SqHcg ttGs1nDDuZYEYFooKKxOsLOWISwR7FTQct9tOUAZa+uu/a1uN8RyjwbB2/c6XOJU HR0FYfIAn37yZ3I6pITFrC5cUJ3E2ddDZBjim7jHvWGgQaSb+5fjor1BRKzrVj46 L6nxLPLsaQeepJtWGvDr9OzIr4zfovhtSdLXOr7uCNrORk49VGQYETjk4hDenB2I c/OFEFt+XTL5Rf95fTQNj7v04rd80witBTSN5xNEyIqYaduFinbql2gEmTkooqpc cJCkBHpphHY5UJj9fnQPqVKQjRL0kaopWXg+4DGEqPu5Rj+V21Cn9BHad8flo4No jjcwSFjRTqJdfc4uzQ20UMZFOkX01Ak/REO77MfxcywE8nqrRFqC8j8IS6gGKWAN +k7/UjgoOzC0qzmjnYBGBXZH/UgQMPFEcIQ2HJ/o1G7vL9y6MocC2Hjc558YLfT6 +IiDmOrIErxV8oMWkarCoTFdW7aILim8AbCQ/XfUezVE3dNkbXuXf/VoUWrOutni CMwpDTcwdZCvdqnwuE2/nEZcCQUTxyaVUorSERjsDGB3SWuInbYniepNqAPFQr0y yaCBn3tBhnKGMIwmRku3 =GYVu -----END PGP SIGNATURE----- From org.python.distutils-sig at pooryorick.com Fri Jun 5 18:00:01 2015 From: org.python.distutils-sig at pooryorick.com (Poor Yorick) Date: Fri, 5 Jun 2015 16:00:01 +0000 Subject: [Distutils] The role and functionality of customize_compiler Message-ID: <20150605160000.GA659@web106.webfaction.com> It looks like the defaults for ccompiler.executables that are set in compiler-specific sublcasses like UnixCCompiler are always overridden by customize_compiler in sysconfig.py. For example, in UnixCcompiler the value for executables['compiler'] is ["cc"], but this will never be used because comtomize_compiler invariably grabs the compiler value out of the Python Makefile and uses that. Should customize_compiler be reorganized a bit so that defaults are actually used if customize_compiler comes up empty-handed for certain values? Next, it looks like customize_compiler is really a method of UnixCcompiler disguised as an independent function. Would it make sense for Ccompiler to have an abstract method, "customize", and to move the current customize_compiler function into UnixCcompiler? On the topic of include directories, would it make sense to add a "sysinclude_dirs" option to the Extension class in order separate out the two different uses that include_dirs currently plays in setup.py scripts? -- Yorick From robertc at robertcollins.net Tue Jun 9 21:09:54 2015 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 10 Jun 2015 07:09:54 +1200 Subject: [Distutils] local platform tags and pip wheel caching Message-ID: So the pip wheel caching feature has promoted a few latent problems into the limelight. One is that many wheels projects built are not actually as compatible as thought. https://github.com/pypa/pip/issues/2882 One is that wheels don't support writing datafiles outside their environment. https://github.com/pypa/pip/issues/2874 And the one I want to talk about is that we don't generate sufficiently good platform markers on Linux. https://github.com/pypa/pip/issues/2875 That bug describes the scenario of using a wheel cache shared across multiple Linux ABIs - e.g. LXC containers with a bind mounted home dir, or NFS on a cluster. pip 7's implicit wheel cache triggers this situation for people. The OP has suggested that being able to control the platform tag that pip a) looks for and b) instructs wheel to use would let them avoid issues in this situation without having to solve the general problem of 'the universe of Linuxs can't agree on anything'. e.g, in each context they could set a /etc/pip.conf containing [global] wheel_platform = contextname and then pip would use that and instruct built wheels to use that. Further, they not that it can be used to e.g. describe other local policies like 'I want hardened C builds' that are not inferrable from context at all today. But since this affects pip *and* wheel, we're bringing it here for discussion. I'm in favour of it - it fits the existing wheel spec; its opt-in, non-intrusive and solves a real, present problem. [just not the 'how do we get these damn things into PyPI, yet]. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From dholth at gmail.com Tue Jun 9 21:25:24 2015 From: dholth at gmail.com (Daniel Holth) Date: Tue, 09 Jun 2015 19:25:24 +0000 Subject: [Distutils] local platform tags and pip wheel caching In-Reply-To: References: Message-ID: Makes sense to me. Could be an environment variable or bdist_wheel option or both. On Tue, Jun 9, 2015 at 3:10 PM Robert Collins wrote: > So the pip wheel caching feature has promoted a few latent problems > into the limelight. > > One is that many wheels projects built are not actually as compatible > as thought. https://github.com/pypa/pip/issues/2882 > One is that wheels don't support writing datafiles outside their > environment. https://github.com/pypa/pip/issues/2874 > And the one I want to talk about is that we don't generate > sufficiently good platform markers on Linux. > https://github.com/pypa/pip/issues/2875 > > That bug describes the scenario of using a wheel cache shared across > multiple Linux ABIs - e.g. LXC containers with a bind mounted home > dir, or NFS on a cluster. pip 7's implicit wheel cache triggers this > situation for people. > > The OP has suggested that being able to control the platform tag that > pip a) looks for and b) instructs wheel to use would let them avoid > issues in this situation without having to solve the general problem > of 'the universe of Linuxs can't agree on anything'. > > e.g, in each context they could set a /etc/pip.conf containing > [global] > wheel_platform = contextname > > and then pip would use that and instruct built wheels to use that. > > Further, they not that it can be used to e.g. describe other local > policies like 'I want hardened C builds' that are not inferrable from > context at all today. > > But since this affects pip *and* wheel, we're bringing it here for > discussion. > > I'm in favour of it - it fits the existing wheel spec; its opt-in, > non-intrusive and solves a real, present problem. [just not the 'how > do we get these damn things into PyPI, yet]. > > -Rob > > -- > Robert Collins > Distinguished Technologist > HP Converged Cloud > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Jun 9 21:27:06 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 9 Jun 2015 20:27:06 +0100 Subject: [Distutils] local platform tags and pip wheel caching In-Reply-To: References: Message-ID: On 9 June 2015 at 20:09, Robert Collins wrote: > I'm in favour of it - it fits the existing wheel spec; its opt-in, > non-intrusive and solves a real, present problem. [just not the 'how > do we get these damn things into PyPI, yet]. Agreed, it sounds like a reasonable approach. One thought - rather than (ab)using the platform tag, maybe an additional "variant" tag could be used (empty by default)? OTOH, I don't see an obvious disadvantage to just using a custom platform tag. Paul From robertc at robertcollins.net Tue Jun 9 22:35:13 2015 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 10 Jun 2015 08:35:13 +1200 Subject: [Distutils] local platform tags and pip wheel caching In-Reply-To: References: Message-ID: On 10 June 2015 at 07:27, Paul Moore wrote: > On 9 June 2015 at 20:09, Robert Collins wrote: >> I'm in favour of it - it fits the existing wheel spec; its opt-in, >> non-intrusive and solves a real, present problem. [just not the 'how >> do we get these damn things into PyPI, yet]. > > Agreed, it sounds like a reasonable approach. > > One thought - rather than (ab)using the platform tag, maybe an > additional "variant" tag could be used (empty by default)? OTOH, I > don't see an obvious disadvantage to just using a custom platform tag. I don't see a space for that in PEP 427; so that might be an option in future but not compatible with wheel 1.0 AIUI. -Rob -- Robert Collins Distinguished Technologist HP Converged Cloud From p.f.moore at gmail.com Tue Jun 9 22:50:06 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 9 Jun 2015 21:50:06 +0100 Subject: [Distutils] local platform tags and pip wheel caching In-Reply-To: References: Message-ID: On 9 June 2015 at 21:35, Robert Collins wrote: > I don't see a space for that in PEP 427; so that might be an option in > future but not compatible with wheel 1.0 AIUI. Good point. We can revisit this for Wheel 2.0, but practicality beats purity for now so I'm OK with using the platform tag on that basis. Paul From dholth at gmail.com Wed Jun 10 02:05:18 2015 From: dholth at gmail.com (Daniel Holth) Date: Wed, 10 Jun 2015 00:05:18 +0000 Subject: [Distutils] local platform tags and pip wheel caching In-Reply-To: References: Message-ID: The platform tag is quite pure enough for this use case. On Tue, Jun 9, 2015, 4:50 PM Paul Moore wrote: > On 9 June 2015 at 21:35, Robert Collins wrote: > > I don't see a space for that in PEP 427; so that might be an option in > > future but not compatible with wheel 1.0 AIUI. > > Good point. We can revisit this for Wheel 2.0, but practicality beats > purity for now so I'm OK with using the platform tag on that basis. > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jaraco at jaraco.com Wed Jun 10 19:25:20 2015 From: jaraco at jaraco.com (Jason R. Coombs) Date: Wed, 10 Jun 2015 17:25:20 +0000 Subject: [Distutils] Setuptools 18.0b1 available for testing (improved Cython build support). Message-ID: <27ED7585-1CD3-4744-8367-9422F10E03D0@jaraco.com> I?m pleased to announce a fix for improved Cython support per Setuptools #288 (https://bitbucket.org/pypa/setuptools/issue/288). I?ve published a beta release 18.0b1 in the Setuptools project downloads (https://bitbucket.org/pypa/setuptools/downloads/setuptools-18.0b1.zip) for pre-release testing. Release notes are here (https://bitbucket.org/pypa/setuptools/src/4249669040239bff20196b17ccd4d73870d37343/CHANGES.txt#cl-10). Please, if you use any projects with Cython modules, give this install a trial run and report any issues in the Setuptools bug tracker. I hope to release 18.0 in the next few weeks if there aren?t any emergent blockers. Thanks! Jason -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.m.bray at gmail.com Wed Jun 10 22:44:15 2015 From: erik.m.bray at gmail.com (Erik Bray) Date: Wed, 10 Jun 2015 16:44:15 -0400 Subject: [Distutils] Setuptools 18.0b1 available for testing (improved Cython build support). In-Reply-To: <27ED7585-1CD3-4744-8367-9422F10E03D0@jaraco.com> References: <27ED7585-1CD3-4744-8367-9422F10E03D0@jaraco.com> Message-ID: On Wed, Jun 10, 2015 at 1:25 PM, Jason R. Coombs wrote: > I?m pleased to announce a fix for improved Cython support per Setuptools > #288 (https://bitbucket.org/pypa/setuptools/issue/288). I?ve published a > beta release 18.0b1 in the Setuptools project downloads > (https://bitbucket.org/pypa/setuptools/downloads/setuptools-18.0b1.zip) for > pre-release testing. Release notes are here > (https://bitbucket.org/pypa/setuptools/src/4249669040239bff20196b17ccd4d73870d37343/CHANGES.txt#cl-10). > > Please, if you use any projects with Cython modules, give this install a > trial run and report any issues in the Setuptools bug tracker. I hope to > release 18.0 in the next few weeks if there aren?t any emergent blockers. Thanks for this, I'll try it out when I get a chance. Although I don't think I've run into this issue often, as I don't usually put Cython in setup_requires, as I prefer to package sources generated by Cython in the source dist (so users installing from the source dist don't need Cython themselves, and I also get more control over the Cython version used). That said, I've wanted to work out a schema for adding Cython to setup_requires only for source checkouts (but remove it when making a source dist), just as a development convenience. I guess I've just never tried it though...or maybe did and realized it was buggy and forgot about it. Good to know that it *should* work now. Erik From ncoghlan at gmail.com Sat Jun 13 18:42:04 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 14 Jun 2015 02:42:04 +1000 Subject: [Distutils] local platform tags and pip wheel caching In-Reply-To: References: Message-ID: On 10 June 2015 at 10:05, Daniel Holth wrote: > The platform tag is quite pure enough for this use case. Aye, +1 from me as well. Overriding the platform tag entirely also has the benefit of letting you categorically prevent the use of wheels from PyPI when you want to force local from-source builds. I've also been pondering this question a fair bit in the context of wanting to be able to define "RHEL 7 + derivatives" as a platform, and between not only full distro derivatives like CentOS and OEL, but also internal variations like RHEL Atomic, Red Hat Storage and RHEL-for-ARM, trying to *derive* the ABI from OS level info in a generic way gets close to impossible. An explicit platform override would make it possible to simply adopt "EPEL7" as the platform tag for such wheel files, since they'd be the Python level equivalent of the pre-built EPEL RPMs [1] Cheers, Nick. [1] https://fedoraproject.org/wiki/EPEL -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guettliml at thomas-guettler.de Thu Jun 18 15:17:08 2015 From: guettliml at thomas-guettler.de (=?UTF-8?B?VGhvbWFzIEfDvHR0bGVy?=) Date: Thu, 18 Jun 2015 15:17:08 +0200 Subject: [Distutils] easy_install deprecated? Message-ID: <5582C4D4.5060903@thomas-guettler.de> Some packages have docs like this: {{{ Installation ========== easy_install foo or pip install foo }}} Maybe I am too new in the python packaging world, but for my eyes calling easy_install looks deprecated. My goal is to make the python world more friendly for newcomers. Are there still reason to use easy_install and not pip? I would like to make the docs more readable and remove the word "or" from them. Things should be straight forward for newcomers and experts know how to help themselves. What do you think? For people loving details: this mail checks if there is a general consensus. It does not matter which packages I refer to. Regards, Thomas G?ttler -- Thomas Guettler http://www.thomas-guettler.de/ From randy at thesyrings.us Thu Jun 18 16:49:18 2015 From: randy at thesyrings.us (Randy Syring) Date: Thu, 18 Jun 2015 10:49:18 -0400 Subject: [Distutils] easy_install deprecated? In-Reply-To: <5582C4D4.5060903@thomas-guettler.de> References: <5582C4D4.5060903@thomas-guettler.de> Message-ID: <5582DA6E.1090403@thesyrings.us> Others may have a more informed opinion, but the only reason I know of to use easy_install is if you are depending on packages that ship compiled eggs for Windows. pip doesn't install eggs and many Windows machines do not have developer tools setup. If your package doesn't ship C extensions as eggs for Windows (and it probably shouldn't, use a wheel instead), just recommending pip is better IMO. *Randy Syring* Husband | Father | Redeemed Sinner /"For what does it profit a man to gain the whole world and forfeit his soul?" (Mark 8:36 ESV)/ On 06/18/2015 09:17 AM, Thomas G?ttler wrote: > Some packages have docs like this: > > {{{ > Installation > ========== > > easy_install foo > > or > > pip install foo > }}} > > Maybe I am too new in the python packaging world, but > for my eyes calling easy_install looks deprecated. > > My goal is to make the python world more friendly for newcomers. > > Are there still reason to use easy_install and not pip? > > I would like to make the docs more readable and remove > the word "or" from them. Things should be straight forward > for newcomers and experts know how to help themselves. > > What do you think? > > For people loving details: this mail checks if there is a general > consensus. It does not > matter which packages I refer to. > > Regards, > Thomas G?ttler > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Thu Jun 18 19:43:00 2015 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 18 Jun 2015 12:43:00 -0500 Subject: [Distutils] easy_install deprecated? In-Reply-To: <5582C4D4.5060903@thomas-guettler.de> References: <5582C4D4.5060903@thomas-guettler.de> Message-ID: On Jun 18, 2015 8:17 AM, "Thomas G?ttler" wrote: > > Some packages have docs like this: > > {{{ > Installation > ========== > > easy_install foo > > or > > pip install foo > }}} > > Maybe I am too new in the python packaging world, but > for my eyes calling easy_install looks deprecated. > > My goal is to make the python world more friendly for newcomers. > > Are there still reason to use easy_install and not pip? easy_install does not install https://pypi.python.org/pypi/backports.ssl_match_hostname ; so, before Python 2.7.9, easy_install does not check SSL certs correctly: * https://docs.python.org/2/library/ssl.html * http://www.python.org/dev/peps/pep-0466 TLDR definitely use pip w/ less than Python 2.7.9. > > I would like to make the docs more readable and remove > the word "or" from them. Things should be straight forward > for newcomers and experts know how to help themselves. > > What do you think? > > For people loving details: this mail checks if there is a general consensus. It does not > matter which packages I refer to. > > Regards, > Thomas G?ttler > > > > -- > Thomas Guettler http://www.thomas-guettler.de/ > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at simplistix.co.uk Thu Jun 18 20:42:14 2015 From: chris at simplistix.co.uk (Chris Withers) Date: Thu, 18 Jun 2015 19:42:14 +0100 Subject: [Distutils] pip caching docs lie? Message-ID: <55831106.2040400@simplistix.co.uk> Hi All, These two sections: https://pip.pypa.io/en/latest/reference/pip_install.html#caching https://pip.pypa.io/en/latest/reference/pip_install.html#wheel-cache ...imply that downloading packages with pip 7.x+ should create an offline cached egg. Not so: (pip_cache)tweedledee:virtualenvs chris$ pip --version pip 7.0.3 from /Users/chris/virtualenvs/pip_cache/lib/python2.7/site-packages (python 2.7) (pip_cache)tweedledee:virtualenvs chris$ pip install xlwt Collecting xlwt Downloading xlwt-1.0.0-py2.py3-none-any.whl (140kB) 100% |????????????????????????????????| 143kB 730kB/s Installing collected packages: xlwt Successfully installed xlwt-1.0.0 (pip_cache)tweedledee:virtualenvs chris$ ls ~/Library/Caches/pip/ http selfcheck.json (pip_cache)tweedledee:virtualenvs chris$ find ~/Library/Caches/pip/ -name wheels (pip_cache)tweedledee:virtualenvs chris$ Nope, no wheel cache there. What am I missing? Are those docs just plain wrong? cheers, Chris From donald at stufft.io Thu Jun 18 21:56:00 2015 From: donald at stufft.io (Donald Stufft) Date: Thu, 18 Jun 2015 15:56:00 -0400 Subject: [Distutils] pip caching docs lie? In-Reply-To: <55831106.2040400@simplistix.co.uk> References: <55831106.2040400@simplistix.co.uk> Message-ID: On June 18, 2015 at 2:50:12 PM, Chris Withers (chris at simplistix.co.uk) wrote: > > What am I missing? Are those docs just plain wrong? > Do you have the ?wheel? package installed? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA From dmaziuk at bmrb.wisc.edu Thu Jun 18 22:46:34 2015 From: dmaziuk at bmrb.wisc.edu (Dimitri Maziuk) Date: Thu, 18 Jun 2015 15:46:34 -0500 Subject: [Distutils] executable egg & top-level __main__.py Message-ID: <55832E2A.8020601@bmrb.wisc.edu> Hi all, I have the following directory structure: myegg/ pkg1/ __init__.py ... pkg2/ __init__.py ... pkg3/ __init__.py ... setup.py __main__.py If I zip this into myegg.egg and run "python myegg.egg", that runs the top-level __main__.py. If I run "python setup.py bdist_egg" and run the resulting egg, I get "/usr/bin/python: can't find '__main__.py'" in the egg. That is correct: according to "unzip -l" __main__.py is not there. How do I get it included? Or more to the point, how do I make that egg directly executable? -- note that I want it to run a script that isn't in any of the bundled packages (but imports from them). (centos 6 with python 2.6 and python-setuptools-0.6.10 and centos 7 with python 2.7 and python-setuptools-0.9.8) -- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: OpenPGP digital signature URL: From aclark at aclark.net Fri Jun 19 02:50:37 2015 From: aclark at aclark.net (Alex Clark) Date: Thu, 18 Jun 2015 20:50:37 -0400 Subject: [Distutils] executable egg & top-level __main__.py In-Reply-To: <55832E2A.8020601@bmrb.wisc.edu> References: <55832E2A.8020601@bmrb.wisc.edu> Message-ID: On 6/18/15 4:46 PM, Dimitri Maziuk wrote: > Hi all, > > I have the following directory structure: > > myegg/ > pkg1/ > __init__.py > ... > pkg2/ > __init__.py > ... > pkg3/ > __init__.py > ... > setup.py > __main__.py > > If I zip this into myegg.egg and run "python myegg.egg", that runs the > top-level __main__.py. If I run "python setup.py bdist_egg" and run the > resulting egg, I get "/usr/bin/python: can't find '__main__.py'" in the > egg. That is correct: according to "unzip -l" __main__.py is not there. > > How do I get it included? > > Or more to the point, how do I make that egg directly executable? -- > note that I want it to run a script that isn't in any of the bundled > packages (but imports from them). > > (centos 6 with python 2.6 and python-setuptools-0.6.10 and centos 7 with > python 2.7 and python-setuptools-0.9.8) Probably can't answer that without seeing setup.py where there should at least be ``packages = find_packages()``. Also sounds like you may be looking for console_scripts. > > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -- Alex Clark ? http://aclark.net From dholth at gmail.com Fri Jun 19 04:15:21 2015 From: dholth at gmail.com (Daniel Holth) Date: Fri, 19 Jun 2015 02:15:21 +0000 Subject: [Distutils] executable egg & top-level __main__.py In-Reply-To: References: <55832E2A.8020601@bmrb.wisc.edu> Message-ID: You are doing the right thing zipping it up yourself instead of relying on bdist_egg. Eggs are a plugin or package format and are not designed to be directly runnable. There are some utilities to help you make executable zip files: https://docs.python.org/dev/library/zipapp.html https://pex.readthedocs.org/en/latest/ If you find it helpful to create your executable more manually, you could unzip the egg and __main__.py into a directory and then zip everything back up. On Thu, Jun 18, 2015 at 8:51 PM Alex Clark wrote: > On 6/18/15 4:46 PM, Dimitri Maziuk wrote: > > Hi all, > > > > I have the following directory structure: > > > > myegg/ > > pkg1/ > > __init__.py > > ... > > pkg2/ > > __init__.py > > ... > > pkg3/ > > __init__.py > > ... > > setup.py > > __main__.py > > > > If I zip this into myegg.egg and run "python myegg.egg", that runs the > > top-level __main__.py. If I run "python setup.py bdist_egg" and run the > > resulting egg, I get "/usr/bin/python: can't find '__main__.py'" in the > > egg. That is correct: according to "unzip -l" __main__.py is not there. > > > > How do I get it included? > > > > Or more to the point, how do I make that egg directly executable? -- > > note that I want it to run a script that isn't in any of the bundled > > packages (but imports from them). > > > > (centos 6 with python 2.6 and python-setuptools-0.6.10 and centos 7 with > > python 2.7 and python-setuptools-0.9.8) > > > Probably can't answer that without seeing setup.py where there should at > least be ``packages = find_packages()``. Also sounds like you may be > looking for console_scripts. > > > > > > > > > > _______________________________________________ > > Distutils-SIG maillist - Distutils-SIG at python.org > > https://mail.python.org/mailman/listinfo/distutils-sig > > > > -- > Alex Clark ? http://aclark.net > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at simplistix.co.uk Fri Jun 19 09:45:40 2015 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 19 Jun 2015 08:45:40 +0100 Subject: [Distutils] pip caching docs lie? In-Reply-To: References: <55831106.2040400@simplistix.co.uk> Message-ID: <5583C8A4.70607@simplistix.co.uk> > On 18 Jun 2015, at 20:56, Donald Stufft wrote: > >> On June 18, 2015 at 2:50:12 PM, Chris Withers (chris at simplistix.co.uk) wrote: >> >> What am I missing? Are those docs just plain wrong? > > Do you have the ?wheel? package installed? Probably not, but those docs don't mention any wheel package requirement... Chris From dmaziuk at bmrb.wisc.edu Fri Jun 19 16:51:43 2015 From: dmaziuk at bmrb.wisc.edu (Dimitri Maziuk) Date: Fri, 19 Jun 2015 09:51:43 -0500 Subject: [Distutils] executable egg & top-level __main__.py In-Reply-To: References: <55832E2A.8020601@bmrb.wisc.edu> Message-ID: <55842C7F.7020403@bmrb.wisc.edu> On 2015-06-18 21:15, Daniel Holth wrote: ... Eggs are a plugin or package format and are not designed > to be directly runnable. Well, google seems to disagree, but ok. > If you find it helpful to create your executable more manually, you > could unzip the egg and __main__.py into a directory and then zip > everything back up. So, since this is all done by a python script, is there a hook in setuptools/setup.py where I can open the egg zipfile and do stuff to it as teh last step of bdist_egg? Or maybe I'll see if I can trick it with file_finders instead... In this case I'm shipping the script for remote execution, I can configure the workflow engine to ship both egg and scriptfile. It would be a touch more convenient to roll the script inside the egg and ship just one file, but not worth the effort. However, I think runnable egg can be handy for dealing with python's version of dll hell. Thanks Dimitri From dmaziuk at bmrb.wisc.edu Fri Jun 19 17:10:55 2015 From: dmaziuk at bmrb.wisc.edu (Dimitri Maziuk) Date: Fri, 19 Jun 2015 10:10:55 -0500 Subject: [Distutils] executable egg & top-level __main__.py In-Reply-To: References: <55832E2A.8020601@bmrb.wisc.edu> Message-ID: <558430FF.20701@bmrb.wisc.edu> On 2015-06-18 19:50, Alex Clark wrote: > Probably can't answer that without seeing setup.py where there should at > least be ``packages = find_packages()``. Yes there is. Also version and name, that's it: straight from the fine manual. > Also sounds like you may be looking for console_scripts. No. I mean, the documentation is sketchy to put it charitably, but it doesn't look like running "python myegg.egg" will call any of those. It looks like it should call "setuptools.installation" entry point(s) but that one's not quite what I'm after. What I'm looking for is exactly what I asked: I have a __main__.py in the root of my directory tree next to setup.py, I want it included in the egg when I run "python setup.py bdist_egg". Cheers, Dimitri From dholth at gmail.com Fri Jun 19 17:44:41 2015 From: dholth at gmail.com (Daniel Holth) Date: Fri, 19 Jun 2015 15:44:41 +0000 Subject: [Distutils] executable egg & top-level __main__.py In-Reply-To: <558430FF.20701@bmrb.wisc.edu> References: <55832E2A.8020601@bmrb.wisc.edu> <558430FF.20701@bmrb.wisc.edu> Message-ID: Did you try py_modules=["__main__"] in setup.py On Fri, Jun 19, 2015, 11:14 AM Dimitri Maziuk wrote: > On 2015-06-18 19:50, Alex Clark wrote: > > > Probably can't answer that without seeing setup.py where there should at > > least be ``packages = find_packages()``. > > Yes there is. Also version and name, that's it: straight from the fine > manual. > > > Also sounds like you may be looking for console_scripts. > > No. I mean, the documentation is sketchy to put it charitably, but it > doesn't look like running "python myegg.egg" will call any of those. It > looks like it should call "setuptools.installation" entry point(s) but > that one's not quite what I'm after. > > What I'm looking for is exactly what I asked: I have a __main__.py in > the root of my directory tree next to setup.py, I want it included in > the egg when I run "python setup.py bdist_egg". > > Cheers, > Dimitri > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmaziuk at bmrb.wisc.edu Fri Jun 19 18:51:38 2015 From: dmaziuk at bmrb.wisc.edu (Dimitri Maziuk) Date: Fri, 19 Jun 2015 11:51:38 -0500 Subject: [Distutils] executable egg & top-level __main__.py: SOLVED In-Reply-To: References: <55832E2A.8020601@bmrb.wisc.edu> <558430FF.20701@bmrb.wisc.edu> Message-ID: <5584489A.7060000@bmrb.wisc.edu> On 06/19/2015 10:44 AM, Daniel Holth wrote: > Did you try py_modules=["__main__"] in setup.py No, of course not: there is no py_modules (or "py_" - anything) anywhere in the developer's guide webpage so how would I know to try it? That is exactly what I was looking for. Thank you. -- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: OpenPGP digital signature URL: From dholth at gmail.com Fri Jun 19 18:59:40 2015 From: dholth at gmail.com (Daniel Holth) Date: Fri, 19 Jun 2015 16:59:40 +0000 Subject: [Distutils] executable egg & top-level __main__.py: SOLVED In-Reply-To: <5584489A.7060000@bmrb.wisc.edu> References: <55832E2A.8020601@bmrb.wisc.edu> <558430FF.20701@bmrb.wisc.edu> <5584489A.7060000@bmrb.wisc.edu> Message-ID: Sorry. The forgettable py_modules parameter is one reason why distutils is terrible and we should stop using it. No we don't have a general replacement to recommend yet. Just keep in mind that having a top-level __main__, if you install two such packages, one __main__ will clobber the other. On Fri, Jun 19, 2015 at 12:51 PM Dimitri Maziuk wrote: > On 06/19/2015 10:44 AM, Daniel Holth wrote: > > Did you try py_modules=["__main__"] in setup.py > > No, of course not: there is no py_modules (or "py_" - anything) anywhere > in the developer's guide webpage so how would I know to try it? > > That is exactly what I was looking for. > > Thank you. > -- > Dimitri Maziuk > Programmer/sysadmin > BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmaziuk at bmrb.wisc.edu Fri Jun 19 19:19:30 2015 From: dmaziuk at bmrb.wisc.edu (Dimitri Maziuk) Date: Fri, 19 Jun 2015 12:19:30 -0500 Subject: [Distutils] executable egg & top-level __main__.py: SOLVED In-Reply-To: References: <55832E2A.8020601@bmrb.wisc.edu> <558430FF.20701@bmrb.wisc.edu> <5584489A.7060000@bmrb.wisc.edu> Message-ID: <55844F22.5000505@bmrb.wisc.edu> On 06/19/2015 11:59 AM, Daniel Holth wrote: > Sorry. The forgettable py_modules parameter is one reason why distutils is > terrible and we should stop using it. No we don't have a general > replacement to recommend yet. Well, it could be mentioned in the "eggsecutable scripts" section, esp. since what that has now doesn't really parse anyway: """ entry_points = { 'setuptools.installation': [ 'eggsecutable = my_package.some_module:main_func', ] } """ OK, so is "eggsecutable" a keyword or can we call it "rose" instead? It's list so what happens when I put more than one "eggsecutable =" in? Key name implies this will run during installation, that's a bit counter-intuitive (unless it's an error and that does actually run at install and not when you try to execute the egg). And so on. Whereas when you look elsewhere you'll find that a) .egg is a zip file and b) when there's a __main__.py in a zip file, python 2.6+ will execute it. > Just keep in mind that having a top-level __main__, if you install two such > packages, one __main__ will clobber the other. Right, thanks. No, the goal is specifically to not install these packages: just run the egg -- so if anybody wants to install, they're welcome to keep the pieces. That does prompt another question, though: what happens if you install two packages with conflicting 'setuptools.installation'? Or 'console_scripts'? I mean, is the problem really only limited to conflicting __main__.py? -- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: OpenPGP digital signature URL: From p.f.moore at gmail.com Fri Jun 19 21:57:46 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 19 Jun 2015 20:57:46 +0100 Subject: [Distutils] executable egg & top-level __main__.py: SOLVED In-Reply-To: <55844F22.5000505@bmrb.wisc.edu> References: <55832E2A.8020601@bmrb.wisc.edu> <558430FF.20701@bmrb.wisc.edu> <5584489A.7060000@bmrb.wisc.edu> <55844F22.5000505@bmrb.wisc.edu> Message-ID: On 19 June 2015 at 18:19, Dimitri Maziuk wrote: > Well, it could be mentioned in the "eggsecutable scripts" section, esp. > since what that has now doesn't really parse anyway: While I understand what you want, and it sounds like you have found a way to achieve it, Daniel is right - this feature of eggs (and indeed the whole egg format in general) is essentially deprecated nowadays. If you want an installable binary, you should be using wheels, and if you want a single-file executable, you should be using zipapp (the zipapp module is new in 3.5, but zipping up the correct directory structure has been supported for ages). The documentation around eggs, and indeed around distutils/setuptools, has never been wonderful. That's a long-standing problem we're looking to fix, but we almost certainly won't be improving the documentation for "eggsecutable scrips" - it's more likely that we'll *remove* that documentation and leave the feature present solely for backward compatibility. (Note: I do *not* speak as the setuptools maintainer - Jason may well have a different view - but what I say above is my understanding as one of the pip maintainers of the general direction the packaging community is heading). Paul From dmaziuk at bmrb.wisc.edu Fri Jun 19 22:24:30 2015 From: dmaziuk at bmrb.wisc.edu (Dimitri Maziuk) Date: Fri, 19 Jun 2015 15:24:30 -0500 Subject: [Distutils] executable egg & top-level __main__.py: SOLVED In-Reply-To: References: <55832E2A.8020601@bmrb.wisc.edu> <558430FF.20701@bmrb.wisc.edu> <5584489A.7060000@bmrb.wisc.edu> <55844F22.5000505@bmrb.wisc.edu> Message-ID: <55847A7E.50307@bmrb.wisc.edu> On 06/19/2015 02:57 PM, Paul Moore wrote: ... if > you want a single-file executable, you should be using zipapp (the > zipapp module is new in 3.5, but zipping up the correct directory > structure has been supported for ages). You're right, I don't really need the egg format for what I'm doing. However, setuptools + setup.py is a familiar idiom like make + Makefile or ant + build.xml. So add a bdist_zip target and have it output a .zip without egg-info cruft instead of making people like me reinvent the wheel every !@#$ing time. Even if the wheel is as simple as piping glob.glob into a zipfile -- which it never is IRL. -- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: OpenPGP digital signature URL: From p.f.moore at gmail.com Fri Jun 19 22:34:10 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 19 Jun 2015 21:34:10 +0100 Subject: [Distutils] executable egg & top-level __main__.py: SOLVED In-Reply-To: <55847A7E.50307@bmrb.wisc.edu> References: <55832E2A.8020601@bmrb.wisc.edu> <558430FF.20701@bmrb.wisc.edu> <5584489A.7060000@bmrb.wisc.edu> <55844F22.5000505@bmrb.wisc.edu> <55847A7E.50307@bmrb.wisc.edu> Message-ID: On 19 June 2015 at 21:24, Dimitri Maziuk wrote: > So add a bdist_zip target and have it output a .zip without egg-info > cruft instead of making people like me reinvent the wheel every !@#$ing > time. Even if the wheel is as simple as piping glob.glob into a zipfile > -- which it never is IRL. That's not a bad idea. It should be possible to do this as a setuptools extension (much as wheel provides the bdist_wheel command). It should probably be bdist_pyz (and produce a file with the .pyz extension) for PEP 441 compatibility. It could then be distributed as a project on PyPI to save everyone else reinventing the wheel, as you say. Paul From dholth at gmail.com Fri Jun 19 22:43:13 2015 From: dholth at gmail.com (Daniel Holth) Date: Fri, 19 Jun 2015 20:43:13 +0000 Subject: [Distutils] executable egg & top-level __main__.py: SOLVED In-Reply-To: <55847A7E.50307@bmrb.wisc.edu> References: <55832E2A.8020601@bmrb.wisc.edu> <558430FF.20701@bmrb.wisc.edu> <5584489A.7060000@bmrb.wisc.edu> <55844F22.5000505@bmrb.wisc.edu> <55847A7E.50307@bmrb.wisc.edu> Message-ID: Don't worry about it. FYI the .egg-info (or .dist-info) cruft is necessary if you want to use certain setuptools features including console_scripts. Without metadata your module will be importable, but pkg_resources won't be able to tell that your distribution is available. On Fri, Jun 19, 2015 at 4:24 PM Dimitri Maziuk wrote: > On 06/19/2015 02:57 PM, Paul Moore wrote: > > ... if > > you want a single-file executable, you should be using zipapp (the > > zipapp module is new in 3.5, but zipping up the correct directory > > structure has been supported for ages). > > You're right, I don't really need the egg format for what I'm doing. > However, setuptools + setup.py is a familiar idiom like make + Makefile > or ant + build.xml. > > So add a bdist_zip target and have it output a .zip without egg-info > cruft instead of making people like me reinvent the wheel every !@#$ing > time. Even if the wheel is as simple as piping glob.glob into a zipfile > -- which it never is IRL. > > -- > Dimitri Maziuk > Programmer/sysadmin > BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dmaziuk at bmrb.wisc.edu Fri Jun 19 23:33:48 2015 From: dmaziuk at bmrb.wisc.edu (Dimitri Maziuk) Date: Fri, 19 Jun 2015 16:33:48 -0500 Subject: [Distutils] executable egg & top-level __main__.py: SOLVED In-Reply-To: References: <55832E2A.8020601@bmrb.wisc.edu> <558430FF.20701@bmrb.wisc.edu> <5584489A.7060000@bmrb.wisc.edu> <55844F22.5000505@bmrb.wisc.edu> <55847A7E.50307@bmrb.wisc.edu> Message-ID: <55848ABC.4070904@bmrb.wisc.edu> On 06/19/2015 03:43 PM, Daniel Holth wrote: > Don't worry about it. > > FYI the .egg-info (or .dist-info) cruft is necessary if you want to use > certain setuptools features including console_scripts. Without metadata > your module will be importable, but pkg_resources won't be able to tell > that your distribution is available. Even easier then: a bdist_zip target that automagically adds py_modules=["__main__"] and saves the file with extension .pyz instead of .egg. I'm not worried about a few extra bytes and besides I might want to use pkg_resourced data files inside the .pyz some day so I'd rather keep at least that part of it anyway. -- Dimitri Maziuk Programmer/sysadmin BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 190 bytes Desc: OpenPGP digital signature URL: From chris at simplistix.co.uk Mon Jun 22 12:50:49 2015 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 22 Jun 2015 11:50:49 +0100 Subject: [Distutils] picky 0.8 released! Message-ID: <5587E889.4090306@simplistix.co.uk> Hi All, I'm please to announce the first release of picky, a tool for checking versions of packages used pip are as specified in their requirements files. I wrote this tool because it's all too easy to have a requirements.txt file for you project that you think covers all the resources you're using, only to find you've been dragging in extra packages unknowingly, and you now have different versions of those in development and production, causing lots of debugging misery! Using picky is as easy as: $ pip install picky $ echo 'picky==0.8' >> requirements.txt $ picky If you want to update your requirements.txt based on your current environment: $ picky --update Full docs are at http://picky.readthedocs.org/en/latest/. Source control and issue trackers are at https://github.com/Simplistix/picky. Any problems, please ask here or mail me direct! cheers, Chris -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at simplistix.co.uk Mon Jun 22 13:20:19 2015 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 22 Jun 2015 12:20:19 +0100 Subject: [Distutils] pip freeze not reporting pip or wsgiref Message-ID: <5587EF73.5040609@simplistix.co.uk> Hi All, A couple of buglets I've noticed while using picky in anger: 1. pip freeze doesn't include a line for pip itself, why is that? 2. pip freeze doesn't include packages provided by the standard libary (ie: wsgiref), even if they're explicitly specified in requirements.txt. What's the pip developers view of both of the above? Is this still the right forum to discuss these things? cheers, Chris From chris.jerdonek at gmail.com Mon Jun 22 18:18:00 2015 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Mon, 22 Jun 2015 09:18:00 -0700 Subject: [Distutils] pip freeze not reporting pip or wsgiref In-Reply-To: <5587EF73.5040609@simplistix.co.uk> References: <5587EF73.5040609@simplistix.co.uk> Message-ID: On Mon, Jun 22, 2015 at 4:20 AM, Chris Withers wrote: > Hi All, > > A couple of buglets I've noticed while using picky in anger: > > 1. pip freeze doesn't include a line for pip itself, why is that? > > 2. pip freeze doesn't include packages provided by the standard libary (ie: > wsgiref), even if they're explicitly specified in requirements.txt. > > What's the pip developers view of both of the above? Is this still the right > forum to discuss these things? There is some discussion about both of these issues here: #1570: "pip freeze/list should filter out 'argparse' for python > 2.6" https://github.com/pypa/pip/issues/1570 #1610: Do not exclude legitimately installed packages from pip freeze https://github.com/pypa/pip/issues/1610 --Chris From fungi at yuggoth.org Mon Jun 22 18:30:36 2015 From: fungi at yuggoth.org (Jeremy Stanley) Date: Mon, 22 Jun 2015 16:30:36 +0000 Subject: [Distutils] pip freeze not reporting pip or wsgiref In-Reply-To: <5587EF73.5040609@simplistix.co.uk> References: <5587EF73.5040609@simplistix.co.uk> Message-ID: <20150622163036.GG2731@yuggoth.org> On 2015-06-22 12:20:19 +0100 (+0100), Chris Withers wrote: > A couple of buglets I've noticed while using picky in anger: > > 1. pip freeze doesn't include a line for pip itself, why is that? [...] You might want `pip list` for this instead. The old freeze behavior omits "common" packages like pip and setuptools, but list does not. -- Jeremy Stanley From chris at simplistix.co.uk Mon Jun 22 20:59:04 2015 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 22 Jun 2015 19:59:04 +0100 Subject: [Distutils] pip freeze not reporting pip or wsgiref In-Reply-To: <20150622163036.GG2731@yuggoth.org> References: <5587EF73.5040609@simplistix.co.uk> <20150622163036.GG2731@yuggoth.org> Message-ID: <55885AF8.5030900@simplistix.co.uk> On 22/06/2015 17:30, Jeremy Stanley wrote: > On 2015-06-22 12:20:19 +0100 (+0100), Chris Withers wrote: >> A couple of buglets I've noticed while using picky in anger: >> >> 1. pip freeze doesn't include a line for pip itself, why is that? > [...] > > You might want `pip list` for this instead. The old freeze behavior > omits "common" packages like pip and setuptools, but list does not. That depends... which is going to give output closer to what is seen in requirements.txt? cheers, Chris From randy at thesyrings.us Fri Jun 26 05:59:15 2015 From: randy at thesyrings.us (Randy Syring) Date: Thu, 25 Jun 2015 23:59:15 -0400 Subject: [Distutils] Introducing Wheelhouse Message-ID: <558CCE13.1090307@thesyrings.us> Wheelhouse is a utility to help maintain a wheelhouse. The code for this project is rather basic, but it's the concept that counts. Putting the concept of a wheelhouse into practice has made managing dependencies for our projects across dev, testing and production environments much, much easier. Checkout Keg to see a project which is using a wheelhouse in conjunction with tox to manage dependencies. What is a Wheelhouse? A wheelhouse is a local cache of python packages in wheel format that gets committed with your code to your VCS. When installing packages during continuous integration and production, the wheels in the wheelhouse are used instead of depending on PyPI or some other network location. Advantages: * Wheels are stored in your DVCS bringing further clarity to exactly what packages are needed/expected and how they have changed over time. * CI builds are faster and more consistent. Due to the increased speed of installing wheels from a local cache instead of pulling them from a network location, we can have tox start with a new virtualenv before every run, thereby insuring all dependencies have been specified and installed into the wheelhouse correctly. * Production deployments are similarly fast and consistent. Since the CI and production servers both pull from the same wheelhouse we have higher certainty that our production code is running against the exact same packages that have been tested. * Since wheels are built on development or build machines, the need for development system packages to be installed on production servers is removed. * Targeting forks, development versions, unpublished, and/or private software for production is much easier than setting up & maintaining a private PyPI server like devpi . * Splits the package management process into two distinct steps: 1. Build packages (from various locations, with specified version) and put wheels in the wheelhouse. 2. Install the latest version of a package from the wheelhouse. Links: https://pypi.python.org/pypi/Wheelhouse https://github.com/level12/wheelhouse *Randy Syring* Husband | Father | Redeemed Sinner /"For what does it profit a man to gain the whole world and forfeit his soul?" (Mark 8:36 ESV)/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Fri Jun 26 08:14:57 2015 From: wes.turner at gmail.com (Wes Turner) Date: Fri, 26 Jun 2015 01:14:57 -0500 Subject: [Distutils] Introducing Wheelhouse In-Reply-To: <558CCE13.1090307@thesyrings.us> References: <558CCE13.1090307@thesyrings.us> Message-ID: Thanks! .wheelhouse.ini! /requirements/wheelhouse/ On Thu, Jun 25, 2015 at 10:59 PM, Randy Syring wrote: > Wheelhouse is a utility to help maintain a wheelhouse. > > The code for this project is rather basic, but it's the concept that > counts. Putting the > concept of a wheelhouse into practice has made managing dependencies for > our projects across dev, > testing and production environments much, much easier. > > Checkout Keg to see a project which is > using a wheelhouse in conjunction with tox to manage dependencies. > What is a Wheelhouse? > > A wheelhouse is a local cache of python packages in wheel format that gets > committed with your code to your VCS. When installing packages during > continuous integration and production, the wheels in the wheelhouse are > used instead of depending on PyPI or some other network location. > Advantages: > > - Wheels are stored in your DVCS bringing further clarity to exactly > what packages are needed/expected and how they have changed over time. > - CI builds are faster and more consistent. Due to the increased speed > of installing wheels from a local cache instead of pulling them from a > network location, we can have tox start with a new virtualenv before every > run, thereby insuring all dependencies have been specified and installed > into the wheelhouse correctly. > - Production deployments are similarly fast and consistent. Since the > CI and production servers both pull from the same wheelhouse we have higher > certainty that our production code is running against the exact same > packages that have been tested. > - Since wheels are built on development or build machines, the need > for development system packages to be installed on production servers is > removed. > - Targeting forks, development versions, unpublished, and/or private > software for production is much easier than setting up & maintaining a > private PyPI server like devpi . > - Splits the package management process into two distinct steps: > 1. Build packages (from various locations, with specified version) > and put wheels in the wheelhouse. > 2. Install the latest version of a package from the wheelhouse. > > Links: > > https://pypi.python.org/pypi/Wheelhouse > https://github.com/level12/wheelhouse > > *Randy Syring* > Husband | Father | Redeemed Sinner > > > *"For what does it profit a man to gain the whole world and forfeit his > soul?" (Mark 8:36 ESV)* > > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Fri Jun 26 08:15:23 2015 From: wes.turner at gmail.com (Wes Turner) Date: Fri, 26 Jun 2015 01:15:23 -0500 Subject: [Distutils] Introducing Wheelhouse In-Reply-To: References: <558CCE13.1090307@thesyrings.us> Message-ID: *wheelhouse.ini On Fri, Jun 26, 2015 at 1:14 AM, Wes Turner wrote: > Thanks! .wheelhouse.ini! /requirements/wheelhouse/ > > On Thu, Jun 25, 2015 at 10:59 PM, Randy Syring > wrote: > >> Wheelhouse is a utility to help maintain a wheelhouse. >> >> The code for this project is rather basic, but it's the concept that >> counts. Putting the >> concept of a wheelhouse into practice has made managing dependencies for >> our projects across dev, >> testing and production environments much, much easier. >> >> Checkout Keg to see a project which is >> using a wheelhouse in conjunction with tox to manage dependencies. >> What is a Wheelhouse? >> >> A wheelhouse is a local cache of python packages in wheel format that >> gets committed with your code to your VCS. When installing packages during >> continuous integration and production, the wheels in the wheelhouse are >> used instead of depending on PyPI or some other network location. >> Advantages: >> >> - Wheels are stored in your DVCS bringing further clarity to exactly >> what packages are needed/expected and how they have changed over time. >> - CI builds are faster and more consistent. Due to the increased >> speed of installing wheels from a local cache instead of pulling them from >> a network location, we can have tox start with a new virtualenv before >> every run, thereby insuring all dependencies have been specified and >> installed into the wheelhouse correctly. >> - Production deployments are similarly fast and consistent. Since the >> CI and production servers both pull from the same wheelhouse we have higher >> certainty that our production code is running against the exact same >> packages that have been tested. >> - Since wheels are built on development or build machines, the need >> for development system packages to be installed on production servers is >> removed. >> - Targeting forks, development versions, unpublished, and/or private >> software for production is much easier than setting up & maintaining a >> private PyPI server like devpi . >> - Splits the package management process into two distinct steps: >> 1. Build packages (from various locations, with specified version) >> and put wheels in the wheelhouse. >> 2. Install the latest version of a package from the wheelhouse. >> >> Links: >> >> https://pypi.python.org/pypi/Wheelhouse >> https://github.com/level12/wheelhouse >> >> *Randy Syring* >> Husband | Father | Redeemed Sinner >> >> >> *"For what does it profit a man to gain the whole world and forfeit his >> soul?" (Mark 8:36 ESV)* >> >> >> _______________________________________________ >> Distutils-SIG maillist - Distutils-SIG at python.org >> https://mail.python.org/mailman/listinfo/distutils-sig >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From justin.uang at gmail.com Sat Jun 27 20:52:15 2015 From: justin.uang at gmail.com (Justin Uang) Date: Sat, 27 Jun 2015 18:52:15 +0000 Subject: [Distutils] Accessing package data files in wheels Message-ID: Hi, What is the recommended way to get access to a data file in my package if I'm using a wheel? pkg_resources seems like it's mainly useful because eggs might be used in a zipped form, but for wheels, I'm guaranteed that the data will be unpacked into a directory structure right? In that case, should I just use __file__ to manually find the location of the file? I'm assuming I can still use pkg_resources, but it feels like I'm adding an unnecessary dependency on setuptools, given that setuptools isn't needed during installation but only when building the wheel. Thanks! Justin -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Sun Jun 28 03:48:00 2015 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 28 Jun 2015 11:48:00 +1000 Subject: [Distutils] Accessing package data files in wheels References: Message-ID: <85r3owsiwf.fsf@benfinney.id.au> Justin Uang writes: > What is the recommended way to get access to a data file in my package > if I'm using a wheel? pkg_resources seems like it's mainly useful > because eggs might be used in a zipped form, but for wheels, I'm > guaranteed that the data will be unpacked into a directory structure > right? In that case, should I just use __file__ to manually find the > location of the file? An advantage of querying package resources using the ?pkg_resources? API is that their lockation *doesn't* need to be known by your program. Let the installation process put them in a sensible location for the system, and discover that location at run-time using ?pkg_resources?. > I'm assuming I can still use pkg_resources, but it feels like I'm > adding an unnecessary dependency on setuptools, given that setuptools > isn't needed during installation but only when building the wheel. You'll be doing the recipient a favour by allowing the package resources to be in a different place from the executable files. -- \ ?Philosophy is questions that may never be answered. Religion | `\ is answers that may never be questioned.? ?anonymous | _o__) | Ben Finney From p.f.moore at gmail.com Sun Jun 28 14:02:40 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 28 Jun 2015 13:02:40 +0100 Subject: [Distutils] Accessing package data files in wheels In-Reply-To: References: Message-ID: On 27 June 2015 at 19:52, Justin Uang wrote: > What is the recommended way to get access to a data file in my package if > I'm using a wheel? pkg_resources seems like it's mainly useful because eggs > might be used in a zipped form, but for wheels, I'm guaranteed that the data > will be unpacked into a directory structure right? In that case, should I > just use __file__ to manually find the location of the file? If you want to avoid a dependency on pkg_resources, you can use pkgutil.get_data (from the stdlib). It doesn't have as many features as pkg_resources, but it does the job in straightforward cases. Regarding zipped form, you are never guaranteed that your code is on the filesystem. Wheels always install unzipped, as you say, but deployment utilities like pyzzer/zipapp, or py2exe/cx_Freeze, bundle your application into a zip file for execution. So unless you want to document that your package doesn't support such deployment methods, you shouldn't assume you're installed to the filesystem (and hence you should avoid using __file__ and doing path manipulation on it). Paul From makoto at makoto.io Mon Jun 29 01:31:30 2015 From: makoto at makoto.io (Makoto) Date: Sun, 28 Jun 2015 19:31:30 -0400 Subject: [Distutils] Requesting the removal of an empty / abandoned PyPi package Message-ID: <559083D2.8040503@makoto.io> Hello, I was told to send an e-mail to this address to request the removal of what appears to be a completely empty and abandoned PyPi package, so that I may make use of the package name. This is the package in question: https://pypi.python.org/pypi/typewriter/0.0.1 This is the project I wish to submit to PyPi, which also uses the name "typewriter", https://github.com/FujiMakoto/python-typewriter (It's really just a single very simple module that adds a simple typewriter animation to print, but I can't think of any other real uses for the package name "typewriter" and even if it is really simple it seems like something that would be useful to others.) My PyPi username is " FujiMakoto" if that is needed. -------------- next part -------------- An HTML attachment was scrubbed... URL: From pmiscml at gmail.com Mon Jun 29 08:56:04 2015 From: pmiscml at gmail.com (Paul Sokolovsky) Date: Mon, 29 Jun 2015 09:56:04 +0300 Subject: [Distutils] Accessing package data files in wheels In-Reply-To: References: Message-ID: <20150629095604.438d6d6d@x230> Hello, On Sun, 28 Jun 2015 13:02:40 +0100 Paul Moore wrote: > On 27 June 2015 at 19:52, Justin Uang wrote: > > What is the recommended way to get access to a data file in my > > package if I'm using a wheel? pkg_resources seems like it's mainly > > useful because eggs might be used in a zipped form, but for wheels, > > I'm guaranteed that the data will be unpacked into a directory > > structure right? In that case, should I just use __file__ to > > manually find the location of the file? > > > If you want to avoid a dependency on pkg_resources, you can use > pkgutil.get_data (from the stdlib). It doesn't have as many features > as pkg_resources, but it does the job in straightforward cases. Which makes everyone in the audience wonder: how it happens that it's 2015, Python is at 3.5, but pkgutil.get_data() is in stdlib, while pkg_resources.resource_stream() isn't? An implementation of pkgutil.get_data() would be based on pkg_resources.resource_stream(), or would contain just the same code as the latter, so it could easily be exposed, and yet it isn't. [] -- Best regards, Paul mailto:pmiscml at gmail.com From ncoghlan at gmail.com Mon Jun 29 09:47:52 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 29 Jun 2015 17:47:52 +1000 Subject: [Distutils] Accessing package data files in wheels In-Reply-To: <20150629095604.438d6d6d@x230> References: <20150629095604.438d6d6d@x230> Message-ID: On 29 June 2015 at 16:56, Paul Sokolovsky wrote: > Hello, > > On Sun, 28 Jun 2015 13:02:40 +0100 > Paul Moore wrote: > >> On 27 June 2015 at 19:52, Justin Uang wrote: >> > What is the recommended way to get access to a data file in my >> > package if I'm using a wheel? pkg_resources seems like it's mainly >> > useful because eggs might be used in a zipped form, but for wheels, >> > I'm guaranteed that the data will be unpacked into a directory >> > structure right? In that case, should I just use __file__ to >> > manually find the location of the file? >> >> >> If you want to avoid a dependency on pkg_resources, you can use >> pkgutil.get_data (from the stdlib). It doesn't have as many features >> as pkg_resources, but it does the job in straightforward cases. > > Which makes everyone in the audience wonder: how it happens that it's > 2015, Python is at 3.5, but pkgutil.get_data() is in stdlib, while > pkg_resources.resource_stream() isn't? An implementation of > pkgutil.get_data() would be based on pkg_resources.resource_stream(), > or would contain just the same code as the latter, so it could easily > be exposed, and yet it isn't. This has the same 3 part answer as a lot of "Why isn't this in the standard library?" questions: 1. nobody has volunteered to drive the standardisation process 2. most of the folks who currently need this functionality want to support older versions of Python, so something pip-installable is actually more use to them than standard library support 3. the folks that *are* currently working on improving the out-of-the-box experience are working on other aspects of that problem (1) is the actual reason, while (2) and (3) are then a couple of the second order factors contributing to (1) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From vinay_sajip at yahoo.co.uk Mon Jun 29 10:00:52 2015 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 29 Jun 2015 08:00:52 +0000 (UTC) Subject: [Distutils] Accessing package data files in wheels In-Reply-To: <20150629095604.438d6d6d@x230> References: <20150629095604.438d6d6d@x230> Message-ID: <77210777.2269405.1435564852388.JavaMail.yahoo@mail.yahoo.com> From: Paul Sokolovsky > Which makes everyone in the audience wonder: how it happens that it's > 2015, Python is at 3.5, but pkgutil.get_data() is in stdlib, while> pkg_resources.resource_stream() isn't? An implementation of > pkgutil.get_data() would be based on pkg_resources.resource_stream(), > or would contain just the same code as the latter, so it could easily >be exposed, and yet it isn't. Perhaps because it's not always that way around - pkg_resources.get_stream_resource, in relevant cases, returns a BytesIO which wraps a byte string. If you want a stream, you could just as easily do this yourself by calling io.BytesIO(pkg_util.get_data('foo.pkg', 'foo_resource')). In the case of file resources only,?pkg_resources.get_stream_resource can open the file and return the stream directly. But this is an optimisation and is not always necessarily available for all different loader implementations - which is perhaps why a `pkgutil.get_data_stream` is not provided. Regards, Vinay Sajip ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Jun 29 10:15:57 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 29 Jun 2015 09:15:57 +0100 Subject: [Distutils] Accessing package data files in wheels In-Reply-To: <20150629095604.438d6d6d@x230> References: <20150629095604.438d6d6d@x230> Message-ID: On 29 June 2015 at 07:56, Paul Sokolovsky wrote: >> If you want to avoid a dependency on pkg_resources, you can use >> pkgutil.get_data (from the stdlib). It doesn't have as many features >> as pkg_resources, but it does the job in straightforward cases. > > Which makes everyone in the audience wonder: how it happens that it's > 2015, Python is at 3.5, but pkgutil.get_data() is in stdlib, while > pkg_resources.resource_stream() isn't? An implementation of > pkgutil.get_data() would be based on pkg_resources.resource_stream(), > or would contain just the same code as the latter, so it could easily > be exposed, and yet it isn't. In addition to Nick's response, which are the main reasons, there is also a more fundamental issue behind this. The PEP 302 definition of a loader only provides a get_data method, which corresponds directly to pkgutil.get_data. Any additional features provided by pkg_resources are not supported directly by the loader protocol, and so could not be guaranteed to be present for an arbitrary loader. pkg_resources provides the extended features (I believe) by special-casing filesystem and zip loaders, and providing an extension mechanism for other loaders to participate in the functionality, but that extension mechanism is not in the stdlib either. So adding more resource loading features means extending the PEP 302 protocols, etc. That's the work that no-one is currently doing that blocks the process. Having said all this, PEP 302 is pretty old now, and importlib makes all of this *much* easier, so (as long as you're only targeting recent Python versions, which stdlib support would be) it's a lot simpler to do this now than it was when we wrote PEP 302. And of course, in practical terms filesystem and zip loaders are the only significant ones that exist anyway... Paul From pmiscml at gmail.com Mon Jun 29 11:26:54 2015 From: pmiscml at gmail.com (Paul Sokolovsky) Date: Mon, 29 Jun 2015 12:26:54 +0300 Subject: [Distutils] Accessing package data files in wheels In-Reply-To: References: <20150629095604.438d6d6d@x230> Message-ID: <20150629122654.53759175@x230> Hello, On Mon, 29 Jun 2015 09:15:57 +0100 Paul Moore wrote: > On 29 June 2015 at 07:56, Paul Sokolovsky wrote: > >> If you want to avoid a dependency on pkg_resources, you can use > >> pkgutil.get_data (from the stdlib). It doesn't have as many > >> features as pkg_resources, but it does the job in straightforward > >> cases. > > > > Which makes everyone in the audience wonder: how it happens that > > it's 2015, Python is at 3.5, but pkgutil.get_data() is in stdlib, > > while pkg_resources.resource_stream() isn't? An implementation of > > pkgutil.get_data() would be based on > > pkg_resources.resource_stream(), or would contain just the same > > code as the latter, so it could easily be exposed, and yet it isn't. > > In addition to Nick's response, which are the main reasons, there is > also a more fundamental issue behind this. > > The PEP 302 definition of a loader only provides a get_data method, > which corresponds directly to pkgutil.get_data. Thanks for this clarification, I expected it to be not just purely logistical reason ("nobody yet got to it"), but also technical/API limitation ("Python core/stdlib doesn't have required prerequisites"). But then it's another level of the same question: we have distutils-sig group with people who oversee Python packaging needs (and related questions), they lately told us (Python community) that e.g. we should stop using Eggs and start using Wheels, so there's lot of active work happens in the area, and yet we're stuck at the old base PEPs which overlooked providing stream access protocol for package resources access. Let that be rhetoric question then, and let everyone assume that so far trading eggs for wheels was more important than closing a visible accidental gap in the stdlib/loader API. > Any additional > features provided by pkg_resources are not supported directly by the > loader protocol, and so could not be guaranteed to be present for an > arbitrary loader. pkg_resources provides the extended features (I > believe) by special-casing filesystem and zip loaders, and providing > an extension mechanism for other loaders to participate in the > functionality, but that extension mechanism is not in the stdlib > either. > > So adding more resource loading features means extending the PEP 302 > protocols, etc. That's the work that no-one is currently doing that > blocks the process. > > Having said all this, PEP 302 is pretty old now, and importlib makes > all of this *much* easier, so (as long as you're only targeting recent > Python versions, which stdlib support would be) it's a lot simpler to > do this now than it was when we wrote PEP 302. And of course, in > practical terms filesystem and zip loaders are the only significant > ones that exist anyway... There was recent discussion on python-dev how other languages are cooler because they allow to create self-contained executables. Python has all parts of the equation already - e.g. the standard way to put Python source inside an executable is using frozen modules. So, that's another usecase for accessing package resources - was support for frozen modules was implemented at all? Granted, frozen modules may be not the easiest way for users to achieve self-contained executables, but the whole point is that Python already has it. I'm worked on another small under-popular scripting language before, and felt that I had all opportunities to make the most perfect language ever - except that it yet needs to be done. As soon as opportunity allowed, I switched to MicroPython, to reuse wealth of APIs and schemas devised by smart people, 90% of which are close to perfect. Well, there's gaps and warts still. So, per the above logic, I don't try to invent something new, but use (at least for starters) frozen modules in MicroPython. But then there's interesting situation - frozen modules are core Python feature, while accessing arbitrary files in them as stream is not. So, even if the core interface doesn't require me to provide stream access to files in frozen modules, I still need to provide it, otherwise I simply won't be able to freeze real-world Python packages. > > Paul -- Best regards, Paul mailto:pmiscml at gmail.com From p.f.moore at gmail.com Mon Jun 29 12:52:00 2015 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 29 Jun 2015 11:52:00 +0100 Subject: [Distutils] Accessing package data files in wheels In-Reply-To: <20150629122654.53759175@x230> References: <20150629095604.438d6d6d@x230> <20150629122654.53759175@x230> Message-ID: On 29 June 2015 at 10:26, Paul Sokolovsky wrote: > and yet we're stuck at the old base PEPs which > overlooked providing stream access protocol for package resources > access. The PEP did not "overlook" stream access. Rather, the compatibility constraints and the need to support existing code meant that we needed to ensure that we required the minimal possible interface from loaders. Even get_data was an optional interface. In practice, many of the constraints around at the time no longer apply, and zip and filesystem loaders remain the most common examples, so the conservative approach of PEP 302 can be revisited (as I said). But someone needs to step up and manage such a change before it will happen. > Let that be rhetoric question then, and let everyone assume that so > far trading eggs for wheels was more important than closing a visible > accidental gap in the stdlib/loader API. The egg->wheel transition was about *distribution* formats. The loader API is a runtime facility. The two are unrelated. One of the problems with eggs was the fact that they were a combined installation and runtime format, so confusing the two aspects is understandable (but still incorrect). > There was recent discussion on python-dev how other languages are > cooler because they allow to create self-contained executables. Python > has all parts of the equation already - e.g. the standard way to put > Python source inside an executable is using frozen modules. So, that's > another usecase for accessing package resources - was support for > frozen modules was implemented at all? See https://docs.python.org/3.5/library/importlib.html#importlib.machinery.FrozenImporter >From that, it appears that the frozen module importer does not implement the ResourceLoader API. So no, get_data is not supported for frozen modules. Of course, you can write your own extension of FrozenImporter for your application, so it's entirely possible to get this to work. But the standard Python bootstrap process (which is FrozenImporter's main use case, AFAIK) doesn't need that feature, which is probably why it's not present. Anyway, as you can see, all the various mechanisms are available, and extending importlib is certainly possible, so as we've said it's really only about someone with the motivation doing the work. It could probably even be done as a 3rd party project in the first place (much like pkg_resources was) and then proposed for inclusion in the stdlib once it has been found to be useful. Paul From brett at python.org Mon Jun 29 16:23:51 2015 From: brett at python.org (Brett Cannon) Date: Mon, 29 Jun 2015 14:23:51 +0000 Subject: [Distutils] Accessing package data files in wheels In-Reply-To: References: <20150629095604.438d6d6d@x230> <20150629122654.53759175@x230> Message-ID: I'm traveling so I can't do a thorough reply, but a goal of mine for Python 3.6 is finally solve the data access problem for packages based on Donald's importlib.resources proposal as well as pkg_resources to try and learn from previous mistakes. On Mon, Jun 29, 2015, 04:52 Paul Moore wrote: > On 29 June 2015 at 10:26, Paul Sokolovsky wrote: > > and yet we're stuck at the old base PEPs which > > overlooked providing stream access protocol for package resources > > access. > > The PEP did not "overlook" stream access. Rather, the compatibility > constraints and the need to support existing code meant that we needed > to ensure that we required the minimal possible interface from > loaders. Even get_data was an optional interface. > > In practice, many of the constraints around at the time no longer > apply, and zip and filesystem loaders remain the most common examples, > so the conservative approach of PEP 302 can be revisited (as I said). > But someone needs to step up and manage such a change before it will > happen. > > > Let that be rhetoric question then, and let everyone assume that so > > far trading eggs for wheels was more important than closing a visible > > accidental gap in the stdlib/loader API. > > The egg->wheel transition was about *distribution* formats. The loader > API is a runtime facility. The two are unrelated. > > One of the problems with eggs was the fact that they were a combined > installation and runtime format, so confusing the two aspects is > understandable (but still incorrect). > > > There was recent discussion on python-dev how other languages are > > cooler because they allow to create self-contained executables. Python > > has all parts of the equation already - e.g. the standard way to put > > Python source inside an executable is using frozen modules. So, that's > > another usecase for accessing package resources - was support for > > frozen modules was implemented at all? > > See > https://docs.python.org/3.5/library/importlib.html#importlib.machinery.FrozenImporter > > From that, it appears that the frozen module importer does not > implement the ResourceLoader API. So no, get_data is not supported for > frozen modules. Of course, you can write your own extension of > FrozenImporter for your application, so it's entirely possible to get > this to work. But the standard Python bootstrap process (which is > FrozenImporter's main use case, AFAIK) doesn't need that feature, > which is probably why it's not present. > > Anyway, as you can see, all the various mechanisms are available, and > extending importlib is certainly possible, so as we've said it's > really only about someone with the motivation doing the work. It could > probably even be done as a 3rd party project in the first place (much > like pkg_resources was) and then proposed for inclusion in the stdlib > once it has been found to be useful. > > Paul > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG at python.org > https://mail.python.org/mailman/listinfo/distutils-sig > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Jun 30 10:01:23 2015 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 30 Jun 2015 18:01:23 +1000 Subject: [Distutils] Accessing package data files in wheels In-Reply-To: References: <20150629095604.438d6d6d@x230> <20150629122654.53759175@x230> Message-ID: On 29 June 2015 at 20:52, Paul Moore wrote: > On 29 June 2015 at 10:26, Paul Sokolovsky wrote: >> and yet we're stuck at the old base PEPs which >> overlooked providing stream access protocol for package resources >> access. > > The PEP did not "overlook" stream access. Rather, the compatibility > constraints and the need to support existing code meant that we needed > to ensure that we required the minimal possible interface from > loaders. Even get_data was an optional interface. > > In practice, many of the constraints around at the time no longer > apply, and zip and filesystem loaders remain the most common examples, > so the conservative approach of PEP 302 can be revisited (as I said). > But someone needs to step up and manage such a change before it will > happen. And active import system experts are even thinner on the ground than packaging experts :) It's good to hear Brett's planning to dive into this for 3.6, though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From idalerali at hotmail.com Mon Jun 29 13:00:57 2015 From: idalerali at hotmail.com (Daler Ali) Date: Mon, 29 Jun 2015 11:00:57 +0000 Subject: [Distutils] =?utf-8?q?how_to_install_pip_and_six_module/packages_?= =?utf-8?q?in_python_3=2E4=2E3?= Message-ID: Sent from Windows Mail -------------- next part -------------- An HTML attachment was scrubbed... URL: