Mailman 3 April 2015 - Distutils-SIG

Can now compile simple extensions
by Greg Ward 29 Apr '18

29 Apr '18

Hi all -- at long last, I found the time to hack in the ability to compile extension modules to the Distutils. Mainly, this meant adding a 'build_ext' command which uses a CCompiler instance for all its dirty work. I also had to add a few methods to CCompiler (and, of course, UnixCCompiler) to make this work. And I added a new module, 'spawn', which takes care of running sub-programs more efficiently and robustly (no shell involved) than os.system. That's needed, obviously, so we can run the compiler! If you're in the mood for grubbing over raw source code, then get the latest from CVS or download a current snapshot. See http://www.python.org/sigs/distutils-sig/implementation.html for a link to the code snapshot. I'm still waiting for more subclasses of CCompiler to appear. At the very least, we're going to need MSVCCompiler to build extensions on Windows. Any takers? Also, someone who knows the Mac, and how to run compilers programmatically there, will have to figure out how to write a Mac-specific concrete CCompiler subclass. The spawn module also needs a bit of work to be portable. I suspect that _win32_spawn() (the intended analog to my _posix_spawn()) will be easy to implement, if it even needs to go in a separate function at all. It looks from the Python Library documentation for 1.5.2 that the os.spawnv() function is all we need, but it's a bit hard to figure out just what's needed. Windows wizards, please take a look at the 'spawn()' function and see if you can make it work on Windows. As for actually compiling extensions: well, if you can figure out the build_ext command, go ahead and give it a whirl. It's a bit cryptic right now, since there's no documentation and no example setup.py. (I have a working example at home, but it's not available online.) If you feel up to it, though, see if you can read the code and figure out what's going on. I'm just hoping *I'll* be able to figure out what's going on when I get back from the O'Reilly conference next week... ;-) Enjoy -- Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Two fixes (at last)
by Greg Ward 29 Apr '18

29 Apr '18

Hi all -- at long last, I have fixed two problems that a couple people noticed a while ago: * I folded in Amos Latteier's NT patches almost verbatim -- just changed an `os.path.sep == "/"' to `os.name == "posix"' and added some comments bitching about the inadequacy of the current library installation model (I think this is Python's fault, but for now Distutils is slavishly aping the situation in Python 1.5.x) * I fixed the problem whereby running "setup.py install" without doing anything else caused a crash (because 'build' hadn't yet been run). Now, the 'install' command automatically runs 'build' before doing anything; to make this bearable, I added a 'have_run' dictionary to the Distribution class to keep track of which commands have been run. So now not only are command classes singletons, but their 'run' method can only be invoked once -- both restrictions enforced by Distribution. The code is checked into CVS, or you can download a snapshot at http://www.python.org/sigs/distutils-sig/distutils-19990607.tar.gz Hope someone (Amos?) can try the new version under NT. Any takers for Mac OS? BTW, all parties involved in the Great "Where Do We Install Stuff?" Debate should take a good, hard look at the 'set_final_options()' method of the Install class in distutils/install.py; this is where all the policy decisions about where to install files are made. Currently it apes the Python 1.5 situation as closely as I could figure it out. Obviously, this is subject to change -- I just don't know to *what* it will change! Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Package versioning
by Timothy Docker 29 Apr '18

29 Apr '18

Hi all, I've been aware that the distutils sig has been simmerring away, but until recently it has not been directly relevant to what I do. I like the look of the proposed api, but have one question. Will this support an installed system that has multiple versions of the same package installed simultaneously? If not, then this would seem to be a significant limitation, especially when dependencies between packages are considered. Assuming it does, then how will this be achieved? I am presently managing this with a messy arrangement of symlinks. A package is installed with its version number in it's name, and a separate directory is created for an application with links from the unversioned package name to the versioned one. Then I just set the pythonpath to this directory. A sample of what the directory looks like is shown below. I'm sure there is a better solution that this, and I'm not sure that this would work under windows anyway (does windows have symlinks?). So, has this SIG considered such versioning issues yet? Cheers, Tim -------------------------------------------------------------- Tim Docker timd(a)macquarie.com.au Quantative Applications Division Macquarie Bank -------------------------------------------------------------- qad16:qad $ ls -l lib/python/ total 110 drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1 -r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1 dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8 drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1 dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1 dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7 dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1 qad16:qad $ ls -l lib/python/1.1 total 30 lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1 lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8 lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1 lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1 lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3 lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7 lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1

3 2

[Distutils-sig] distutils charter and interscript
by John Skaller 29 Apr '18

29 Apr '18

3 2

Versioned trove classifiers for Django
by James Bennett 06 Apr '17

06 Apr '17

Following up on some IRC discussion with other folks: There is precedent (Plone) for PyPI trove classifiers corresponding to particular versions of a framework. So I'd like to get feedback on the idea of expanding that, particularly in the case of Django. The rationale here is that the ecosystem of Django-related packages is quite large, but -- as I know all too well from a project I'm working on literally at this moment -- it can be difficult to ensure that all of one's dependencies are compatible with the version of Django one happens to be using. Adding trove classifier support at the level of individual versions of Django would, I think, greatly simplify this: tools could easily analyze which packages are compatible with an end user's chosen version, there'd be far less manual guesswork, etc., and the rate of creation of new classifiers would be relatively low (we tend to have one X.Y release/year or thereabouts, and that's the level of granularity needed). Assuming there's consensus around the idea of doing this, what would be the correct procedure for getting such classifiers set up and maintained?

5 17

Beyond wheels 1.0: helping downstream, FHS and more
by David Cournapeau 22 May '15

22 May '15

Hi there, During pycon, Nick mentioned there was interest in updating the wheel format to support downstream distributions. Nick mentioned Linux distributions, but I would like to express interest for other kind of downstream distributors like Anaconda from Continuum or Canopy from Enthought (disclaimer: I work for Enthought). Right now, wheels have the following limitations for us: 1. lack of post/pre install/removing 2. more fine-grained installation scheme 3. lack of clarify on which tags vendors should use for custom wheels: some packages we provide would not be installable on "normal" python, and it would be nice to have a scheme to avoid confusion there as well. At least 1. and 2. are of interest not just for us. Regarding 2., it looks like anything in the <wheel_name>.data/data directory will be placed as is in sys.prefix by pip. This is how distutils scheme is defined ATM, but I am not sure whether that's by design or accident ? I would suggest to use something close to autotools, with some tweaks to work well on windows. I implemented something like this in my project bento ( https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfi…), but we could of course tweak that. For 1., I believe it was a conscious decision not to include them in wheel 1.0 ? Would it make sense to start a discussion to add it to wheel ? I will be at the pycon sprints until wednesday evening, so that we can flesh some concrete proposal first, if there is enough interest. As a background: at Enthought, we have been using eggs to distribute binaries of python packages and other packages (e.g. C libraries, compiled binaries, etc...) for a very long time. We had our own extensions to the egg format to support this, but I want to get out of eggs so as to make our own software more compatible with where the community is going. I would also like to avoid making ad-hoc extensions to wheels for our own purposes. thanks, David

18 109

Beyond wheel 1.0: more fine-grained installation scheme
by David Cournapeau 13 May '15

13 May '15

Hi, I am splitting up the previous thread into one thread / proposal to focus the discussion. Assuming the basis of this proposal does not sound too horrible, I would make a proof of concept in a pip branch, so that we can flush out the details and then write an actual spec (I guess an updated wheel format would need a new PEP ?). The goal of this thread is to flush out a more fine-grained installation scheme, so that wheels can install files anywhere they want (at least within sys.prefix/sys.exec_prefix). I see two issues: 1. defining what the scheme should be 2. how should it be implemented in wheel: there are different trade-offs depending on whether we want this feature to be part of wheel format 1.* or 2.0. First, my understanding of the current situation: * per the wheel PEP 427, anything in the wheel top directory and not in distribution-1.0.data is installed in site-package * every top directory in distribution-1.0.data/ needs to be mapped to the scheme as defined in distutils install command. * pip rejects any directory in distribution-1.0.data/ which is not in the scheme from 2. My suggestion for a better scheme would be to use an extended version of the various default directories defined by autotools. The extension would handle windows-specifics. More concretely: # Suggested variables The goal of supporting those variables is to take something that is flexible enough to support almost any installation scheme, without putting additional burden on the developer. People who do not want/need the flexibility will not need to do anything more than what they do today. The variables I would suggest are every variable defined in https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfi…, except for destdir which is not relevant here. On unix, the defaults follow autotools, and on windows, I mapped almost everything relative to sys.exec_prefix, except for the bindir/sbindir/libexecdir which map to "$prefix\Scripts". The $sitedir variable would need to be updated to use the value from distutils instead of the hardcoded value I put in that file as well. # How to handle the augmented scheme Right now, if I wanted to install something in say $prefix/share/doc, I would need to put it in distribution-1.0.data/data/share/doc, but this prevents use from handling different platforms differently. OTOH, this is the only way I can see to make the new scheme backward compatible with pip versions who would not understand the new scheme. I don't have a good sense of what we should do there, the core pip team may have a better sense. For now, I would be happy to just make a proof of concept not caring about backward compatibility in a pip branch. Does that sound like a workable basis to flush out an actual proposal ? thanks, David

4 5

Query about serial module installation
by Shamsudheen Padath 29 Apr '15

29 Apr '15

Hi, I have been trying to communicate through my usb port through as part of my project work. For this I have usb to serial converter installed in my system. I am working on python 3.4.3 and installed the serial module pyserial-2.7.win32_py3k.exe <file:///C:/Users/shamsu/Desktop/hari/pyserial-2.7.win32_py3k.exe> downloaded from www.pyhton.org/. My OS is 64 bit MS Windows 8.1. The trial programme i have been trying is import serial ser = serial.Serial(5) print (ser.portstr) ser.write("hello") ser.close() and this produces an error message COM6 Traceback (most recent call last): File "C:\Users\shamsu\Desktop\ff.py", line 4, in <module> ser.write("hello") # write a string File "C:\Python34\lib\site-packages\serial\serialwin32.py", line 283, in write data = to_bytes(data) File "C:\Python34\lib\site-packages\serial\serialutil.py", line 76, in to_bytes b.append(item) # this one handles int and str for our emulation and ints for Python 3.x TypeError: an integer is required As a newer to the Python i couldn't what the problem is and got stuck on it. Expecting a solution to my problem. Shamsudheen p

3 2

cross-compilation patches
by Václav Šmilauer 27 Apr '15

27 Apr '15

Hi evrybody, I hope this is the right SIG for cross-compilation of Python interpreter (not of 3rd party modules -- yet). As you may know, cross-compilation was being declared unsupported for a long time and contributions in that area were not always gladly accepted. There have been numerous attempts to improve the situation, usually ending in patches in bugzilla which quickly became obsolete, resulting in frustration and great and repeated waste of manpower. Most active proponents of these patches were, it seems, Tarek Ziadé and Roumen Petrov. Roumen made an impressive set of patches against 3.4 at the end of 2013, all referenced from [1]; they are split into modernization of mingw&cygwin classes (as a prominent case of cross-building), cross-building interpreter core, cross-building core modules, and cross-installation. Each of these references individual issues (e.g. interpreter core in [2] references 15 issues, each with patch, core modules [3] have 24 sub-issues). Since I have been needing cross-build support (build for Windows, and recently also Xeon Phi as targets, on Linux host) myself, I would like these patches not fall under the table again. I was pointed to this SIG to see what could be done (or, more precisely, what could I do) for those patches to be progressively integrated. Some people mention that the infrastructure is currently missing a cross-building bot, without which regressions will soon creep in again. What can be done in this regard, what is needed? Cheers, Václav --- [1] http://bugs.python.org/issue3871#msg199695 [2] http://bugs.python.org/issue17605 [3] http://bugs.python.org/issue18653

1 0

Re: [Distutils] Proper handling of PEP420 namespace packages with setuptools and pip
by M.-A. Lemburg 25 Apr '15

25 Apr '15

[adding list back on CC] On 22.04.2015 16:11, Christoph Schmitt wrote: > Am 2015-04-22 12:59, schrieb M.-A. Lemburg: >> On 22.04.2015 12:38, Robert Collins wrote: >>> On 22 April 2015 at 22:13, Christoph Schmitt <dev-mailings(a)gongy.de> wrote: >>>> Hello again, >>>> >>>> since I haven't got any replies yet I'm trying to make myself a bit more >>>> precise now. I consider the behaviour described in my original posting a >>>> bug. I posted to this list because the setuptools docs say "Please use the >>>> distutils-sig mailing list [3] for questions and discussion about >>> >>>> Test 3) >>>> DO NOT delcare namespace_packages=['coolpkg'] in setup.py of each project >>>> Result: all modules can be imported >>> >>> This is correct AFAICT. >>> >>> the setuptools namespace_packages thing predates PEP-420, and because >>> PEP-420 namespaces don't interoperate with .pth file based packages >>> (expecially when you get into interactions between system non-PEP-420 >>> + virtualenv PEP-420 packages!) changing this is super hard: you'll >>> guarantee to break many existing installs. >>> >>> Perhaps there should be a new keyword, but since nothing is needed to >>> make things work, it seems like it would be rather redundant. >> >> You can make use of the namespace_packages keyword argument to setup() >> optional depending on which Python version is running it. >> >> I guess that's the only way forward unless you want to break >> the package for previous Python versions. >> >> However, doing so may be hard for namespaces which are used >> by a lot of packages. >> >> Perhaps setuptools could simply ignore the keyword for >> Python 3.3+ and then rely on PEP 420 to get things working >> in more or less the same way: >> >> https://www.python.org/dev/peps/pep-0420/ > > I would be fine with declaring namespace_packages conditionally. But doesn't this affect sdist in > another way than install (or pip install)? If an sdist intended for use with Python < 3.3 is created > with Python >= 3.3, the included metadata (egg-info) would look different (I don't know if pip > relies on egg-info or setup.py). The egg-info generated by setup.py at install time :-) > That would apply also if namespace_packages would be ignored > automatically for Python >= 3.3 as you proposed. > > As a consequence, two distributions were neccessary. One with namespace_packages declared and > containing an __init__.py (with pkg_resources.declare_namespace) and another one without those > additions. But how does setuptools figure out to leave out the __init__.py for non-declard namespace > packages in the latter case? Like I mentioned above: it's probably better for setuptools to handle this in a consistent way rather than changing all setup.pys. One detail I'm not sure about is how compatible the two namespace package techniques really are. The PEP 420 hand waves over possible differences and only addresses pkgutil, not the setuptools approach, which is by far more common. For most simply applications not relying on any of the advanced features, they will most likely be compatible. I guess some experiments are necessary to figure that out. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 22 2015) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

7 12