Mailman 3 January 2013 - Distutils-SIG

Merging buildout-versions in buildout
by Chris Withers 11 Jan '13

11 Jan '13

Hi All, My take on this, as the person who may or may not have offered to do the work... After reading all the discussion, it sounds like the approach buildout-versions currently takes is the one that's going to give the most milleage: - print the versions picked for any packages (as opposed to being specified by the buildout), in a format compatible with being pasted into buildout.cfg's versions section. This should happen by default. (when *wouldn't* you want this to be shown?) - have a key that points to a file where the text above is appended. It's up to the use whether this is buildout.cfg or not. For myself, if the buildout is tiny, I put a versions section at the end and point this option at buildout.cfg. Otherwise, I point it as a versions.cfg. I had forgotten how important the interaction between the versions section and the extends mechanism was. I've also made great use of this. While I like the idea of a versions format compatible with pip*, it sounds like that would kill off the extends interaction, which would be bad. I'll note that there appears to be no contention over the optional python-version key to stop you accidentally running a buildout with the wrong python. cheers, Chris * I have to admit to currently thinking hard about migrating away from buildout; its development has been hampered by people with not enough time trying to solve hard problems, the dropping of Windows support (yeah, I hate windows too, but its still used enough that I need to support it, unfortunately I'm not competent enough to help with development *of* buildout on Windows). That said, I don't know that pip/virtualenv would be any better. -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

5 8

zc.buildout 1.7.0 released, the don't upgrade to buildout 2 release
by Jim Fulton 11 Jan '13

11 Jan '13

See http://pypi.python.org/pypi/zc.buildout/1.7.0 Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm

1 0

Better version pinning in buildout (buildout-versions)
by Jim Fulton 11 Jan '13

11 Jan '13

Lots of people agree that buildout-versions us useful and the author has volunteered to contribute it to core buildout. Cool! Of course, as a part of buildout, rather than an extension, it can be streamlined a bit. Also, based on experience, I think we can simplify versions management a bit further. When I implemented the versions mechanism in buildout, I decided to build on the configuration model. This has worked pretty well, especially for complex use cases like layering version specifications (e.g. building on "known good sets"), but this approach has added a little bit of drudgery. Because I didn't want to stomp on existing versions sections, I made the name of the section containing versions require configuration, which means you end up with:: versions = versions [versions] ... I also think it's a little awkward to have buildout update files that are also edited by users. Based on this, I propose that buildout-versions get incorporated into buildout in the following way: 1. New buildout option named ``versions-file`` which takes the name of a file. to contain version information. It is not a configuration file. It is a file consisting of comments (#...) and version specifications:: # whatever foo = 1.3 If it doesn't exist, it wil be created. (I'm not sure it's a good idea to create the file implicitly though.) Any version constraints found in the file are added to the buildout version constraints. Version constraints found in the versions-file override version constraints obtained via a versions option, if any. 2. New buildout option: ``update-versions-file``. If this is true, then any picked/unpinned versions are appended to the versions file and reported in the output. There's a command-line option, ``-V`` to set this to true for a run. It's an error to use this if ``versions-file`` isn't set. 3. The ``allow-picked-versions`` option gets a new allowed value of ``warn``. if there are unpicked versions and this option is set to ``warn``, then picked/unpinned versions are reported. Also, if ``allow-picked-versions`` is true, there will be no error if ``update-versions-file`` is true. 4. New buildout option: ``python-version`` that restricts the Python version, with the same semantics as buildout-version provides now. Thoughts? Did I miss anything? Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm

9 22

pip install --upgrade bug on python3.x
by Christian Tismer 10 Jan '13

10 Jan '13

Hi folks, I recognized a glitch with pip when using the --upgrade option. From the traceback: File "./setuptools/dist.py", line 103 except ValueError, e: ^ SyntaxError: invalid syntax This seems to happen only if --upgrade is used, regardless whether a module was installed before or not. It is easy to circumvent by first uninstalling and then avoiding the option. Should I file an issue? Where? Or submit a patch? cheers - Chris -- Christian Tismer :^) <mailto:tismer@stackless.com> Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

6 8

SandboxViolation: mkdir('/usr/local/virtualenvs/ansible/share', 511) {}
by Guy Rozendorn 10 Jan '13

10 Jan '13

Hi, I'm trying to make ansible (https://github.com/Infinidat/ansible) compatible with easy_install (so I can use it later in places that use easy_install and not pip). When I install it with pip, everything works, but not with easy_install: ~/g/i/ansible git:devel ❯❯❯ easy_install -U ansible (env: ansible) ⬆ ✱ ◼ Searching for ansible Reading http://pypi01.infinidat.com/simple/ansible/ Best match: ansible 0.9-develop-30 Downloading http://pypi01.infinidat.com/media/dists/ansible-0.9-develop-30.tar.gz#md5=1… Processing ansible-0.9-develop-30.tar.gz Writing /var/folders/f9/d108d_ds0sz0_qlnzr25_bcr0000gn/T/easy_install-ThM02g/ansible-0.9-develop-30/setup.cfg Running ansible-0.9-develop-30/setup.py -q bdist_egg --dist-dir /var/folders/f9/d108d_ds0sz0_qlnzr25_bcr0000gn/T/easy_install-ThM02g/ansible-0.9-develop-30/egg-dist-tmp-kZvXdB DATA FILES=[('/usr/local/virtualenvs/ansible/share/ansible/', ['./library/add_host', './library/apt', './library/apt_repository', './library/assemble', './library/async_status', './library/async_wrapper', './library/authorized_key', './library/command', './library/copy', './library/cron', './library/debug', './library/easy_install', './library/ec2', './library/facter', './library/fail', './library/fetch', './library/file', './library/fireball', './library/get_url', './library/git', './library/group', './library/group_by', './library/ini_file', './library/lineinfile', './library/mail', './library/mount', './library/mysql_db', './library/mysql_user', './library/nagios', './library/ohai', './library/pause', './library/ping', './library/pip', './library/postgresql_db', './library/postgresql_user', './library/raw', './library/script', './library/seboolean', './library/selinux', './library/service', './library/setup', './library/shell', './library/slurp', './library/subversion', './library/supervisorctl', './library/svr4pkg', './library/template', './library/user', './library/virt', './library/wait_for', './library/yum'])] warning: no files found matching 'packaging/distutils/setup.py' error: Setup script exited with error: SandboxViolation: mkdir('/usr/local/virtualenvs/ansible/share', 511) {} AFAIK, the violation occurs becuase ansible wants to write stuff to /usr/local/virtualenv/ansible, which is out of the sandbox. According to http://docs.python.org/2/distutils/setupscript.html#installing-additional-f…, data files can be outside of the sandbox ('/etc/init.d'), and it looks like this isn't supported by distribute? Any idea would be greatly appreciated. Thanks, Guy

2 1

GLM analysis questions
by Abduljalil Sireis 09 Jan '13

09 Jan '13

Dear all, I would like to ask questions about structural solution for GLM analysis of fMRI data. I hope it is not a problem for such long description! :) The subjects of our experiment see two types of video clips. One type is dangerous car accidents and the other is normal car advertisements. We have number of subjects perform this task with one session per subject. Each run consists of 6 clips from each type, the two successive spots are separated by black screen and a yellow cross to fix the subjects eyes. There was not a specific order of the spots but more than two successive ones of the same type were not allowed. All videos were accompanied with voice, so we have visual and audio stimulation. Attached with this email a table of onsets,durations and indices of both conditions. TR (scan repeat time) = 3000 ms. Functional measurement consists of 220 scans. Here is a formulation of the steps needed to be done in the analysis: The GLM analysis that needs to be performed to get the beta estimates is a multiple regression analysis. In other words, I need to specify a model that includes predictors (one predictor for each condition, a confound-mean predictor (all-one vector), and optionally head-motion and trend predictors), and then to fit this model to the data to obtain the beta-estimates. In order to run the GLM, I first need to assemble the model. For this, I can use the list containing the onsets and durations and indices of the stimuli. The next step is to create a predictor for each condition. For each condition, I start off with a vector (220 elements, one for each timepoint) of zeros, then I put ones at those timepoints at which that specific condition was presented. After that, I convolve these predictors with the hemodynamic response function (2-gamma or Boynton) to make the predictors resemble the timing and shape of BOLD measurements. I can add additional predictors to the model, i.e. nuisance predictors that model out effects of motion and slowly-varying trends. I also have to include a confound-mean predictor (i.e. an all-one vector) to model the mean level of brain activation. All predictors together form the model. Since the block durations are not multiples of the TR, it is needed to convolve the hemodynamic response function with the cognitive predictors (one for condition 1, and one for condition 2) at millisecond resolution, and then to subsample the result to get the predicted response at the TR resolution (one measurement each 3 s). * **QUESTIONS** * There is a very unclear thing to me relating to predictors. What should I do exactly in multiplying the HRF with the cognitive predictors? I mean, how the cognitive predictors can be created using the data i have? And also the HRF function. Then what are the steps needed to do the subsample? Better also with the sense of nipype, i have already installed the software but never used it before.

1 0

Re: [Distutils] Some clarifications and/or corrections to PEP 376
by Vinay Sajip 08 Jan '13

08 Jan '13

If you move the files, other things can potentially break, too. For example, any scripts installed will have their shebangs adjusted to point to a particular interpreter, e.g. a venv's interpreter. If they get moved to a different location, the shebang lines may become invalid. I don't believe that we have to support such moving of files outside of the knowledge of packaging tools. Regards, Vinay Sajip >________________________________ > From: Kevin Horn <kevin.horn(a)gmail.com> >To: Vinay Sajip <vinay_sajip(a)yahoo.co.uk> >Sent: Monday, 31 December 2012, 19:19 >Subject: Re: [Distutils] Some clarifications and/or corrections to PEP 376 > > >On Mon, Dec 31, 2012 at 12:59 PM, Vinay Sajip <vinay_sajip(a)yahoo.co.uk> wrote: > > >> I agree that PEP 376 isn't ideal in what it specifies in this area. The simplest >>solution would surely be absolute paths only. What are the downsides apart from >>the disk space used for the extra lengths of the filenames? >> >>Regards, >> >>Vinay Sajip >> >> >> > > >If you ever move the files, the absolute paths won't be valid any more. It seems kind of unlikely that this would cause problems, but it might. > >-- >Kevin Horn > >

7 14

Some clarifications and/or corrections to PEP 376
by Paul Moore 07 Jan '13

07 Jan '13

I've been looking at the details of PEP 376 (Database of Installed Python Distributions) and there are a couple of aspects that I don't think work properly alongside PEP 370 (Per-user site-packages directory).. 1. The dist-info directory for a distribution is stated as being "located in the site-packages directory". It's not clear how this is intended to work in a PEP 370 world with multiple site-packages. What I propose is that the description be changed to be worded in terms of sysconfig-style locations: the dist-info directory is located in whichever of purelib or platlib is used by the distribution. When the distribution uses both, purelib is preferred, when it uses neither (!) purelib is used. In nearly all cases, this is the same as currently. The exceptions are posix_home (where the directory name isn't "site-packages" but its function is the same), posix_prefix (where purelib and platlib differ, and PEP 376 is currently ambiguous as to which is implied), and any custom schemes that might be created (where PEP 376 is silent, and this proposal has the benefit of at least being specific). I do not believe this changes any actual practice - as far as I have been able to determine any code using dist-info at the moment follows this proposal in the corner cases where it differs from PEP 376. 2. File paths in RECORD are stated as being relative to the "base location" as long as the files sit under that location, or under the "install prefix" where that is above the base location. That's a messy but precise way of saying you can go up a bit as long as you stay within sys.prefix. (Other paths are platform format absolute paths, which is an ugly discrepancy but the only really viable option). Again, this doesn't work as I'd expect it should in a PEP 370 user site-packages. In this case, the answer isn't quite as clear. I would suggest, looking at the schemes defined in sysconfig, allowing any files located in one of the defined sysconfig paths (for the current scheme) to be recorded as relative paths. The paths should be relative to whichever of purelib and platlib was chosen in (1) above. The exception is that where pureXXX and platXXX do not use the same base (posix_prefix and possibly custom schemas) it should not be allowed to have a relative path from a pureXXX to a platXXX or vice versa. That equates to something slightly stricter than the current scheme (although not in a way that's likely to be used in practice), but using the appropriate one of {base} or {userbase} rather than sys.prefix. As noted, the two exceptions are custom schemes (where PEP 376 is silent and would probably end up effectively mandating platform absolute paths throughout) and posix_prefix, where platlib/platinclude/platstdlib are located under {platbase} rather than {base} and the current proposal would probably allow relative paths to be used a little more often that PEP 376 allows. Current practice that I've seen here is not as clear cut, but mostly in the sense that code doesn't consider the corner cases at all, and probably only follows PEP 376 by luck, if at all... Comments would be appreciated. I'm pretty happy that my proposal for (1) is an improvement. With (2) I'm concerned about the posix_prefix case, particularly as I'm a Windows developer and my understanding of Python's layout on POSIX is limited. But I honestly don't think it's possible to do much better than my proposal, and I do think that PEP 376 as it stands now is distinctly worse because it doesn't consider PEP 370 or the posix_prefix case. Also, while these proposals are not 100% backward compatible, I don't believe they change behaviour in the most common scenarios, and given that PEP 376 currently has limited adoption (mainly in distutils2, I believe) I'd suggest that the minor compatibility break now is better than keeping the current system until it's more widely adopted and there are more users to impact. Obviously I'd be looking for views from Tarek and Éric in particular here, as it's their code in distutils2 that would be directly impacted. Paul.

4 10

zc.buildout-1.7.0b2 released
by Jim Fulton 06 Jan '13

06 Jan '13

The main goal of the 1.7 release is to provide an update for buildout 1 that won't upgrade itself to buildout 2. zc.buildout-1.7.0b2 fixes a backward-compatibility problem reported in zc.buildout-1.7.0b1. To get the release, download the new buildout 1 bootstrap.py script from http://downloads.buildout.org/1/bootstrap.py: wget http://downloads.buildout.org/1/bootstrap.py and use it to bootstrap your projects. Because this is a non-final release, you need to use the -t option when you run it: python bootstrap.py -t Please try it and let me know if you have any problems. If I don't hear anything, I'll release 1.7.0 final in a week or so. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm

2 6

buildout.dumppickedversions and buildout_versions
by Jim Fulton 04 Jan '13

04 Jan '13

On Sat, Dec 15, 2012 at 11:56 AM, Chris Withers <chris(a)simplistix.co.uk> wrote: > On 15/12/2012 09:33, Lele Gaifax wrote: >> >> Alex Clark <aclark(a)aclark.net> writes: >> >>> 19:43 < cullerton> Folks, there's something wonky going on with >>> buildout this morning. When I run bootstrap, it creates a buildout >>> script using buildout 2.0. >>> 0a4. This fails with an error (TypeError: get_dist() takes exactly 4 >>> arguments (3 given)). Changing the version to buildout 1.4.4 seems to >>> fix things. >> >> >> FYI, I just met the same error (using 2.0.0a5), and the culprit was the >> buildout.dumppickedversion extension, that monkey-patches the >> easy_install.py::Installer._get_dist() method (it surprised me at first >> that the error message mentions “get_dist()” but at the referenced line >> in easy_install.py there is a call to “_get_dist()”...). It took a pdb >> session and a “p inspect.getmodule(self._get_dist)” to find out what's >> going on. > > > Yeah, the stuff that buildout.dumppickedversions and buildout_versions > provide should just be in the core. > > Jim, if I worked up a pull request would you merge it? Maybe :) I tried to look at buildout.dumppickedversions and buildout_versions and I couldn't quite tell what they were for or how they were different. Are they different? Or are they 2 versions of the same thing? http://pypi.python.org/pypi/buildout.dumppickedversions gives me a fair idea what it does, but http://pypi.python.org/pypi/buildout-versions looks like almost the same thing and http://packages.python.org/buildout_versions/index.html doesn't state what problem it solves. I realize that a tool to help manage version pins would be helpful. I punted on this when I added the versions feature. Before adding this feature to buildout, I'd like a clearer statement of what problems are being solved and what the approach is. Alternatively, I'd be open to adding an API to make writing an extension possible without depending on internal implementation details. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton Jerky is better than bacon! http://zo.pe/Kqm

7 14