Mailman 3 February 2011 - Distutils-SIG

Re: [Distutils] zc.buildout/zc, recipe.egg: passing extra options for build_ext
by kiorky 19 Feb '11

19 Feb '11

Le 19/02/2011 01:45, Maurits van Rees a écrit : > Op 18-02-11 19:29, kiorky schreef: >> >> Le 18/02/2011 17:18, Maurits van Rees a écrit : >>> Hi, >>> >>> I see that you can add specify some custom build settings for packages with >>> C-extensions: >>> http://pypi.python.org/pypi/zc.recipe.egg#creating-eggs-with-extensions-nee… >>> >>> >> You can with minitage.recipe.scripts : >> - http://pypi.python.org/pypi/minitage.recipe.egg#specific-options >> - >> http://pypi.python.org/pypi/minitage.recipe.egg#installing-eggs-with-custom… >> > > Thanks for the info. I had a look at these. I saw that this recipe pulls in > some other minitage packages as well and in the end even pulls in mercurial, > which failed for me on Ubuntu due to a missing bzip2 library. Well... mercurial is an historical dependency from minitage.core. I'm thinking about to remove it or not for years. > > But I'll have a look at how this recipe does its stuff. Prefixing the extra > options with 'bdistext' to avoid confusion with other options at least seems a > good idea. This is also a bit historical but yes i found also that it is more straightforward like that... Think that minitage.recipe.* are mostly 'compatible' with zc.recipe.* stuff. You can even replace them on the fly on your buildout without touching anything but adding 'buildout.minitagificator' to your extensions (it will just monkey patch the recipes and entry points). > > Cheers, > -- Cordialement, KiOrKY GPG Key FingerPrint: 0x1A1194B7681112AF Pensez à l’environnement. N’imprimez ce courriel que si vous en avez vraiment besoin.

1 0

buildout download utility and failed checksums in cache
by Łukasz Nowak 19 Feb '11

19 Feb '11

Hello, Buildout download utility allows to provide md5sum to check if downloaded (or available file from cache) is correct. In case of using download-cache and having wrong file in this cache it is required to manually remove this file and redownload. Is it possible to have redownload "request" inside of download utility? In case if file exists in cache, but passed checksum does not match, download utility cloud redownload this file, re check sum and eventually succeed. Is it ok to have such change of buildout download cache behaviour? Regards, Luke -- Łukasz Nowak IT Specialist email(a)lnowak.com http://lnowak.com/ Skype: Shufla jid: shufla(a)jabster.pl gg: 1157726 ``Use the Source, Luke...''

2 3

zc.buildout/zc, recipe.egg: passing extra options for build_ext
by Maurits van Rees 18 Feb '11

18 Feb '11

Hi, I see that you can add specify some custom build settings for packages with C-extensions: http://pypi.python.org/pypi/zc.recipe.egg#creating-eggs-with-extensions-nee… The options you can pass are limited though. In my case for the InformixDB package I want to pass '--esql-threadlib = posix' to the build process. Currently this is not possible. Is there a reason why only the current hardcoded options (like include-dirs) are available for customizing? It looks like it should be possible to change the zc.recipe.egg code to look for other options and pass them to the build_ext call. The only thing that might be tricky is to make a difference between options in the buildout part that should be passed to build_ext and options that have a totally different meaning and should be ignored by build_ext. Would this be a good addition to zc.recipe egg? I could create a branch and try it. That should be a branch of zc.buildout, right? Or are there other ways that currently already work? My workaround for now will likely be to make a copy of the original source dist tarball of InformixDB and add a setup.cfg in there with this contents: [build_ext] --esql-threadlib = posix From a local test this seems to be working. BTW, I did not know that you could specify these options in setup.cfg; I found that out today by looking through the zc.buildout code. :-) Cheers, -- Maurits van Rees Web App Programmer at Zest Software: http://zestsoftware.nl Personal website: http://maurits.vanrees.org/

2 1

[buildout] including optional python packages for testing
by Chris Withers 18 Feb '11

18 Feb '11

Hi All, I'm using buildout to develop a package that uses sqlalchemy and sqlalchemy-migrate. As a result, I'd like to test with various real database drivers; mysql, postgres, etc. However, I don't want to force anyone who wants to check out and run the tests to have all supported database drivers available and compiled. So, I'm looking for some way of specifying "optional eggs" when running the buildout, and not in buildout.cfg so they don't accidentally get checked in. How can I achieve this? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

3 5

Re: [Distutils] easy_install installing beta version of psycopg2
by P.J. Eby 18 Feb '11

18 Feb '11

At 12:41 PM 2/18/2011 +0000, Daniele Varrazzo wrote: >I've tried to be good and replace the download_url, which used to >refer to a directory containing all the packages, to a direct link to >the package to download (see >http://pypi.python.org/pypi/psycopg2/2.3.2). It wasn't enough as >probably previous versions listed in /simple/ still refer to the >directory (see http://pypi.python.org/simple/psycopg2/). > >Is there a way to clean the list in /simple/, apart from deleting all >the previous releases from PyPI, which would be a net loss for PyPI as >a repository? Yes. Just edit those releases. Log in to PyPI, go to your package page, then click on "Releases". There will be an "edit" link for each release.

2 1

Tools should (at least optionally) prefer final releases. (was Re: easy_install installing beta version of psycopg2)
by Jim Fulton 18 Feb '11

18 Feb '11

On Fri, Feb 18, 2011 at 5:29 AM, Tarek Ziadé <ziade.tarek(a)gmail.com> wrote: > On Fri, Feb 18, 2011 at 10:00 AM, Chris Withers <chris(a)simplistix.co.uk> wrote: >> On 17/02/2011 00:01, Daniele Varrazzo wrote: >>> >>> I don't think it is fair to ask to stop listing the beta from the >>> homepage and the download page of the 2-pages website of the module: >> >> Indeed, but I think it's totally fair to expect users to pin to the version >> they require using whatever package management policy they choose. >> >> Of course, sure, it would be nice if easy_install had a sane default of >> installing only final releases, but what're the chances of PJE implementing >> that? ;-) >> >> What's the "distribute" take on this? > > If someone wants to contribute a "--prefer-final" option to Distribute > --extracted from zc.buildout-- I am fine adding it, and even > activating this behavior by default, so people opt in to get the non > final releases. Thank you for noting the buildout feature, which has been around since 2007. :) I really think tools should support this feature. I also think the default should be to prefer final. I think this is much better than maintaining separate indexes. > > I think it would be a good way to let the user control the behavior of > the installer better, and that's what zc.buildout wants to do next > AFAIK. Yes. buildout2 will prefer final by default. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton

1 0

Re: [Distutils] easy_install installing beta version of psycopg2
by P.J. Eby 18 Feb '11

18 Feb '11

At 12:01 AM 2/17/2011 +0000, Daniele Varrazzo wrote: >On Wed, Feb 16, 2011 at 8:40 PM, P.J. Eby <pje(a)telecommunity.com> wrote: > > At 05:47 PM 2/16/2011 +0000, Daniele Varrazzo wrote: > >> > >> Do I, as a packager, have the possibility to say "what I have specified > >> on PyPI as stable release is exactly what I mean"? > > > > If you don't want easy_install to find it, don't list it on the pages > > referred to in your "Home Page" or "Download URL" on PyPI. Easy_install > > reads those pages because many package authors do not provide directly > > usable URLs or packages on PyPI. > >I'm sorry, it is obvious that I have not spent so much time into this >problem as the designer of this feature. But it still don't get the >rationale behind discarding available, non-ambiguous metadata in >favour of screen scraping. When easy_install was first written, PyPI didn't even support *uploading*. And the quality of available metadata on PyPI is still quite sketchy -- many packages will have only one file uploaded for an outdated version, but still have good downloads on their home pages or download URLs. While I realize this can be an inconvenience for the people who DO have good metadata, the truth is that there are many packages for which "unstable" version numbers are in fact the *best* download choice for many users. Without a uniform versioning system that *every* package adheres to (and PEP 386 isn't actually sufficient for this yet -- something more like "semantic versioning" is needed), there's no way to easily set a policy across projects for "how stable a version do you want to download?" In general, the practice for most projects is to simply publish unstable, "don't download this unless you really mean it" versions via "development" links, such as links to svn or other repositories. These links don't look like downloads to easy_install, except for the #egg=project-version tag that tells it how to interpret them, and saying '#egg=myproject-beta' suffices to ensure that only a specific installation request for 'myproject==beta' will follow the link. (Unfortunately, this tag does *not* currently override easy_install's native interpretation of the link, so tacking '#egg=psycopg-beta' on the end of your download links won't stop them being detected as future versions. See below for other workarounds.) >I don't think it is fair to ask to stop listing the beta from the >homepage and the download page of the 2-pages website of the module: >how should I advertise there is a beta around and testing is welcome? You can make a direct download link, or make the filename not one that easy_install will recognize. For example, if you rename the downloads to "beta-psycopg-whatever.*", or use a URL that redirects, like /beta-download, that's then redirected by the web server to the appropriate file location. easy_install only looks at links that *appear* to be distutils-or-setuptools-generated archives for the desired project. Another alternative is to block easy_install from parsing your home page or download links, by simply providing those links inline in your PyPI project description, and *removing* the specific fields from the current release and all previous releases stored on PyPI. Still another would be to block the 'setuptools' and 'distribute' user agents from your website, returning 404s. >the shortcomings of a package manager Well, technically, this'd be a feature. Granted, it's only a feature for users of projects whose maintainers are *not* keeping a well-groomed PyPI page. ;-) I guess it is a shortcoming in the sense that there ought to be a way to stop it from using this feature. In retrospect, the #egg=proj-ver feature should override easy_install's native interpretation of a URL, rather than just adding to it, and I think I would be justified in considering this a bug worthy of fixing. Unfortunately, even if I fixed that today, it wouldn't have ANY effect on 99% of the field installations of any Python package management tools: there are still people using 4 or 5 year old versions of easy_install, and a lot of people use Distribute (via their OS install), which is a year behind the setuptools trunk on various things. Most other Python package management solutions are based on top of easy_install in one way or another, as well. Pip is the main package manager that uses its own link-finding algorithm, but it only supports source installation AFAIK. Distutils2 uses a link-finding algorithm that was lifted pretty much verbatim from easy_install, though I think there may have been some additions to it since I last looked at it. > > You can prevent this as a package author, by specifying zip_safe=False in > > your setup() script. > >psycopg2 is not zip_safe, as it contains a C extension. C extensions don't make a package not-zip-safe - they just mean that if you install it zipped, there has to be an egg cache. The cache is used to unzip the C extension. If you want to force easy_install to unzip your package from the get-go, then specifying zip_safe=False explicitly is required. (Of course, a user can still download the egg manually or override this on the command line, but that's their problem in that case.)

5 5

Re: [Distutils] easy_install installing beta version of psycopg2
by P.J. Eby 18 Feb '11

18 Feb '11

At 05:47 PM 2/16/2011 +0000, Daniele Varrazzo wrote: >Do I, as a packager, have the possibility to say "what I have specified >on PyPI as stable release is exactly what I mean"? If you don't want easy_install to find it, don't list it on the pages referred to in your "Home Page" or "Download URL" on PyPI. Easy_install reads those pages because many package authors do not provide directly usable URLs or packages on PyPI. >possibly not requiring a writable egg cache (see You can prevent this as a package author, by specifying zip_safe=False in your setup() script.

5 4

zc.buildout: download utility vs. local resources
by Thomas Lotze 17 Feb '11

17 Feb '11

Hi all, I'm currently second-guessing a detail of the zc.buildout's download utility I wrote last year (*) and I want to ask for others' opinions. What I question is how the settings for cache usage and download destination end up determining whether the copy of the resource that client code receives is private to the buildout, or shared. The current logic involves an optimisation that tries to avoid copying files in the file system by creating hard links instead if possible. (**) I'd like to make the decision about using a private or shared copy more explicit and foreseeable. The obvious way to do this is adding another keyword parameter to the download call that expresses whether hard-linking should be attempted. I'm however not at all clear on what would be a sensible default value: Using a shared copy may not be desirable if client code is going to modify the file after downloading, so attempting to create hard links by default is the more dangerous behaviour. Always copying, on the other hand, means that when using a cache (or "downloading" a file-system resource) every download creates multiple copies of potentially large files in the file system, which is unnecessary in what I think is the majority of cases. It would be very helpful if someone could offer some input. Thank you. Thomas (*) The download utility is an API inside zc.buildout that can be used by recipes or zc.buildout itself to download HTTP or file-system resources. It can be configured to use a download cache and to put the downloaded resource in a particular place. In any case, it returns the file-system path of that copy of the downloaded resource which is to be used by the client code. (**) If an HTTP resource is downloaded, * having neither a cache nor a download destination puts the resource at a temporary path unique to the current download call, * using a cache but no download destination results in accessing the (shared) file inside the cache, * if using a download destination but no cache, a private copy of the resource is put in the destination, * if using both a cache and a download destination, the utility tries to hard-link the cached file to the destination and failing that, copies it, so the result may be either a private copy or a shared one. If the resource comes from the file system, * using no download destination results in shared access to either the original or cached resource, * specifying a download destination results in accessing either a shared or private copy of the resource depending on whether hard links from its original or cached path to the download destination are possible.

1 1

zc.buildout: the extended-by option
by Thomas Lotze 17 Feb '11

17 Feb '11

Hi, the buildout section used to have an option named "extended-by" that has been deprecated for quite a while. The option is not (no longer?) present in documentation and tests and the only thing left of it is the implementation and a check that is supposed to result in a deprecation warning. However, there's a bug in the implementation that looks as if it was caused by some refactoring, which would cause a NameError to be raised instead of the deprecation warning being printed. To me, this looks as if it was safe to just remove any traces of the option, including the buggy implementation. The only downside would be that configurations which still use the option would fail silently instead of raising the NameError. I don't consider this to be of much importance in practice, but before going ahead and removing code, I wanted to give people a chance to disagree. Thank you. -- Thomas

2 2