Le 19/02/2011 01:45, Maurits van Rees a écrit :
> Op 18-02-11 19:29, kiorky schreef:
>> Le 18/02/2011 17:18, Maurits van Rees a écrit :
>>> I see that you can add specify some custom build settings for packages with
>> You can with minitage.recipe.scripts :
>> - http://pypi.python.org/pypi/minitage.recipe.egg#specific-options
> Thanks for the info. I had a look at these. I saw that this recipe pulls in
> some other minitage packages as well and in the end even pulls in mercurial,
> which failed for me on Ubuntu due to a missing bzip2 library.
Well... mercurial is an historical dependency from minitage.core.
I'm thinking about to remove it or not for years.
> But I'll have a look at how this recipe does its stuff. Prefixing the extra
> options with 'bdistext' to avoid confusion with other options at least seems a
> good idea.
This is also a bit historical but yes i found also that it is more
straightforward like that...
Think that minitage.recipe.* are mostly 'compatible' with zc.recipe.* stuff. You
can even replace them on the fly on your buildout without touching anything but
adding 'buildout.minitagificator' to your extensions (it will just monkey patch
the recipes and entry points).
GPG Key FingerPrint: 0x1A1194B7681112AF
Pensez à l’environnement.
N’imprimez ce courriel que si vous en avez vraiment besoin.
Buildout download utility allows to provide md5sum to check if
downloaded (or available file from cache) is correct.
In case of using download-cache and having wrong file in this cache it
is required to manually remove this file and redownload.
Is it possible to have redownload "request" inside of download utility?
In case if file exists in cache, but passed checksum does not match,
download utility cloud redownload this file, re check sum and
Is it ok to have such change of buildout download cache behaviour?
Łukasz Nowak IT Specialist email(a)lnowak.com http://lnowak.com/
Skype: Shufla jid: shufla(a)jabster.pl gg: 1157726
``Use the Source, Luke...''
I see that you can add specify some custom build settings for packages
The options you can pass are limited though. In my case for the
InformixDB package I want to pass '--esql-threadlib = posix' to the
build process. Currently this is not possible.
Is there a reason why only the current hardcoded options (like
include-dirs) are available for customizing? It looks like it should be
possible to change the zc.recipe.egg code to look for other options and
pass them to the build_ext call. The only thing that might be tricky is
to make a difference between options in the buildout part that should be
passed to build_ext and options that have a totally different meaning
and should be ignored by build_ext.
Would this be a good addition to zc.recipe egg? I could create a branch
and try it. That should be a branch of zc.buildout, right?
Or are there other ways that currently already work?
My workaround for now will likely be to make a copy of the original
source dist tarball of InformixDB and add a setup.cfg in there with this
--esql-threadlib = posix
From a local test this seems to be working.
BTW, I did not know that you could specify these options in setup.cfg; I
found that out today by looking through the zc.buildout code. :-)
Maurits van Rees
Web App Programmer at Zest Software: http://zestsoftware.nl
Personal website: http://maurits.vanrees.org/
I'm using buildout to develop a package that uses sqlalchemy and
As a result, I'd like to test with various real database drivers; mysql,
However, I don't want to force anyone who wants to check out and run the
tests to have all supported database drivers available and compiled.
So, I'm looking for some way of specifying "optional eggs" when running
the buildout, and not in buildout.cfg so they don't accidentally get
How can I achieve this?
Simplistix - Content Management, Batch Processing & Python Consulting
At 12:41 PM 2/18/2011 +0000, Daniele Varrazzo wrote:
>I've tried to be good and replace the download_url, which used to
>refer to a directory containing all the packages, to a direct link to
>the package to download (see
>http://pypi.python.org/pypi/psycopg2/2.3.2). It wasn't enough as
>probably previous versions listed in /simple/ still refer to the
>directory (see http://pypi.python.org/simple/psycopg2/).
>Is there a way to clean the list in /simple/, apart from deleting all
>the previous releases from PyPI, which would be a net loss for PyPI as
Yes. Just edit those releases. Log in to PyPI, go to your package
page, then click on "Releases". There will be an "edit" link for each release.
On Fri, Feb 18, 2011 at 5:29 AM, Tarek Ziadé <ziade.tarek(a)gmail.com> wrote:
> On Fri, Feb 18, 2011 at 10:00 AM, Chris Withers <chris(a)simplistix.co.uk> wrote:
>> On 17/02/2011 00:01, Daniele Varrazzo wrote:
>>> I don't think it is fair to ask to stop listing the beta from the
>>> homepage and the download page of the 2-pages website of the module:
>> Indeed, but I think it's totally fair to expect users to pin to the version
>> they require using whatever package management policy they choose.
>> Of course, sure, it would be nice if easy_install had a sane default of
>> installing only final releases, but what're the chances of PJE implementing
>> that? ;-)
>> What's the "distribute" take on this?
> If someone wants to contribute a "--prefer-final" option to Distribute
> --extracted from zc.buildout-- I am fine adding it, and even
> activating this behavior by default, so people opt in to get the non
> final releases.
Thank you for noting the buildout feature, which has been around since 2007. :)
I really think tools should support this feature. I also think the default
should be to prefer final. I think this is much better than maintaining separate
> I think it would be a good way to let the user control the behavior of
> the installer better, and that's what zc.buildout wants to do next
Yes. buildout2 will prefer final by default.
At 12:01 AM 2/17/2011 +0000, Daniele Varrazzo wrote:
>On Wed, Feb 16, 2011 at 8:40 PM, P.J. Eby <pje(a)telecommunity.com> wrote:
> > At 05:47 PM 2/16/2011 +0000, Daniele Varrazzo wrote:
> >> Do I, as a packager, have the possibility to say "what I have specified
> >> on PyPI as stable release is exactly what I mean"?
> > If you don't want easy_install to find it, don't list it on the pages
> > referred to in your "Home Page" or "Download URL" on PyPI. Easy_install
> > reads those pages because many package authors do not provide directly
> > usable URLs or packages on PyPI.
>I'm sorry, it is obvious that I have not spent so much time into this
>problem as the designer of this feature. But it still don't get the
>rationale behind discarding available, non-ambiguous metadata in
>favour of screen scraping.
When easy_install was first written, PyPI didn't even support
*uploading*. And the quality of available metadata on PyPI is still
quite sketchy -- many packages will have only one file uploaded for
an outdated version, but still have good downloads on their home
pages or download URLs.
While I realize this can be an inconvenience for the people who DO
have good metadata, the truth is that there are many packages for
which "unstable" version numbers are in fact the *best* download
choice for many users. Without a uniform versioning system that
*every* package adheres to (and PEP 386 isn't actually sufficient for
this yet -- something more like "semantic versioning" is needed),
there's no way to easily set a policy across projects for "how stable
a version do you want to download?"
In general, the practice for most projects is to simply publish
unstable, "don't download this unless you really mean it" versions
via "development" links, such as links to svn or other
repositories. These links don't look like downloads to easy_install,
except for the #egg=project-version tag that tells it how to
interpret them, and saying '#egg=myproject-beta' suffices to ensure
that only a specific installation request for 'myproject==beta' will
follow the link.
(Unfortunately, this tag does *not* currently override easy_install's
native interpretation of the link, so tacking '#egg=psycopg-beta' on
the end of your download links won't stop them being detected as
future versions. See below for other workarounds.)
>I don't think it is fair to ask to stop listing the beta from the
>homepage and the download page of the 2-pages website of the module:
>how should I advertise there is a beta around and testing is welcome?
You can make a direct download link, or make the filename not one
that easy_install will recognize. For example, if you rename the
downloads to "beta-psycopg-whatever.*", or use a URL that redirects,
like /beta-download, that's then redirected by the web server to the
appropriate file location. easy_install only looks at links that
*appear* to be distutils-or-setuptools-generated archives for the
Another alternative is to block easy_install from parsing your home
page or download links, by simply providing those links inline in
your PyPI project description, and *removing* the specific fields
from the current release and all previous releases stored on
PyPI. Still another would be to block the 'setuptools' and
'distribute' user agents from your website, returning 404s.
>the shortcomings of a package manager
Well, technically, this'd be a feature. Granted, it's only a feature
for users of projects whose maintainers are *not* keeping a
well-groomed PyPI page. ;-) I guess it is a shortcoming in the
sense that there ought to be a way to stop it from using this
feature. In retrospect, the #egg=proj-ver feature should override
easy_install's native interpretation of a URL, rather than just
adding to it, and I think I would be justified in considering this a
bug worthy of fixing.
Unfortunately, even if I fixed that today, it wouldn't have ANY
effect on 99% of the field installations of any Python package
management tools: there are still people using 4 or 5 year old
versions of easy_install, and a lot of people use Distribute (via
their OS install), which is a year behind the setuptools trunk on
various things. Most other Python package management solutions are
based on top of easy_install in one way or another, as well.
Pip is the main package manager that uses its own link-finding
algorithm, but it only supports source installation
AFAIK. Distutils2 uses a link-finding algorithm that was lifted
pretty much verbatim from easy_install, though I think there may have
been some additions to it since I last looked at it.
> > You can prevent this as a package author, by specifying zip_safe=False in
> > your setup() script.
>psycopg2 is not zip_safe, as it contains a C extension.
C extensions don't make a package not-zip-safe - they just mean that
if you install it zipped, there has to be an egg cache. The cache is
used to unzip the C extension. If you want to force easy_install to
unzip your package from the get-go, then specifying zip_safe=False
explicitly is required. (Of course, a user can still download the
egg manually or override this on the command line, but that's their
problem in that case.)
At 05:47 PM 2/16/2011 +0000, Daniele Varrazzo wrote:
>Do I, as a packager, have the possibility to say "what I have specified
>on PyPI as stable release is exactly what I mean"?
If you don't want easy_install to find it, don't list it on the pages
referred to in your "Home Page" or "Download URL" on
PyPI. Easy_install reads those pages because many package authors do
not provide directly usable URLs or packages on PyPI.
>possibly not requiring a writable egg cache (see
You can prevent this as a package author, by specifying
zip_safe=False in your setup() script.
I'm currently second-guessing a detail of the zc.buildout's
download utility I wrote last year (*) and I want to ask for
What I question is how the settings for cache usage and download
destination end up determining whether the copy of the resource
that client code receives is private to the buildout, or shared.
The current logic involves an optimisation that tries to avoid
copying files in the file system by creating hard links instead
if possible. (**)
I'd like to make the decision about using a private or shared
copy more explicit and foreseeable. The obvious way to do this is
adding another keyword parameter to the download call that
expresses whether hard-linking should be attempted. I'm however
not at all clear on what would be a sensible default value:
Using a shared copy may not be desirable if client code is going
to modify the file after downloading, so attempting to create
hard links by default is the more dangerous behaviour. Always
copying, on the other hand, means that when using a cache (or
"downloading" a file-system resource) every download creates
multiple copies of potentially large files in the file system,
which is unnecessary in what I think is the majority of cases.
It would be very helpful if someone could offer some input.
(*) The download utility is an API inside zc.buildout that can be
used by recipes or zc.buildout itself to download HTTP or
file-system resources. It can be configured to use a download
cache and to put the downloaded resource in a particular place.
In any case, it returns the file-system path of that copy of the
downloaded resource which is to be used by the client code.
(**) If an HTTP resource is downloaded,
* having neither a cache nor a download destination puts the
resource at a temporary path unique to the current download
* using a cache but no download destination results in accessing
the (shared) file inside the cache,
* if using a download destination but no cache, a private copy of
the resource is put in the destination,
* if using both a cache and a download destination, the utility
tries to hard-link the cached file to the destination and
failing that, copies it, so the result may be either a private
copy or a shared one.
If the resource comes from the file system,
* using no download destination results in shared access to
either the original or cached resource,
* specifying a download destination results in accessing either a
shared or private copy of the resource depending on whether
hard links from its original or cached path to the download
destination are possible.
the buildout section used to have an option named "extended-by" that has
been deprecated for quite a while. The option is not (no longer?) present
in documentation and tests and the only thing left of it is the
implementation and a check that is supposed to result in a deprecation
warning. However, there's a bug in the implementation that looks as if it
was caused by some refactoring, which would cause a NameError to be raised
instead of the deprecation warning being printed.
To me, this looks as if it was safe to just remove any traces of the
option, including the buggy implementation. The only downside would be
that configurations which still use the option would fail silently
instead of raising the NameError. I don't consider this to be of much
importance in practice, but before going ahead and removing code, I wanted
to give people a chance to disagree. Thank you.