There's been a previous post (Tarek, "Fixing the mess in sdist/egg_info")
about the fact that setuptools fails due to a circular dependency:
distutils.sdist.run() -> setuptools.build_py.data_files ->
setuptools.egg_info.run() -> distutils.sdist.add_defaults() ->
setuptools.build_py.data_files -> etc
Actually I'm working packaging sphinx and it fails for this reason. Has
anything moved to fix this problem?
Regards,
Antonio
At 04:00 PM 5/18/2009 -0400, Jean-Paul Calderone wrote:
>Hello,
>
>I've just noticed that when using setuptools, the name of the file created
>by the sdist command changes. The version passed to setup is "0.9.33+r17283"
>but the file written is "Nevow-0.9.33-r17283.tar.gz".
>
>This change causes various problems for me. How can I avoid it?
You would need to subclass the sdist command, or use a
post-processing step to rename it.
Hello,
I've just noticed that when using setuptools, the name of the file created
by the sdist command changes. The version passed to setup is "0.9.33+r17283"
but the file written is "Nevow-0.9.33-r17283.tar.gz".
This change causes various problems for me. How can I avoid it?
Thanks,
Jean-Paul
Hi,
Since we are talking about versioning, here is one more complaint
which may fall into the bugreport category or not, I am not sure. There
are some version checks in some codepaths which do not make much sense
IMHO, in particular:
- distutils checks the versions of the toolchain. This is broken. If
I want to use gcc built from svn to build my extensions, distutils
should just do what I ask it to do, and not barf as it currently does.
- bdist_msi also fails to run when the project version is not a
released version. I can't see the rationale for that behavior, and it is
not consistent with others bdist_ anyway.
thanks,
David
Hello
I am making a separate email for this topic to make sure no one misses it.
There are *many* benefits of adding entry points into Distutils. In
fact, adding a plugin system in there,
will help make the commands more extendable and therefore will help us
in the long term to remove things
out of Distutils.
So any strong opinion against them ?
If not, I'll add a section in PEP 376 - and the first use case I can
see is having a pluggable way to extend
the list of files contained in .egg-info.
The work to be done would be to extract entry points from
pkg_resources (the discovery work that consists of looking
for entry_points.txt files will be covered by the APIs already
described in that PEP) This is not hard.
Cheers
Tarek
--
Tarek Ziadé | http://ziade.org
At 11:38 PM 5/14/2009 +0200, Tarek Ziadé wrote:
>That happened because I got a bit lost in the way finders were working
>in setuptools, which seemed
>a bit over-engineered at that time to me (but I might end up changing
>my mind at sime point when I understand it better).
The finders mechanism is there to allow support for arbitrary PEP 302
importers -- i.e., sys.path strings that don't go to the filesystem.
The main reason why there are two finders currently (find_on_path and
find_in_zip) is that find_in_zip supports egg directory
bundling. That is, an .egg zipfile may contain arbitrarily nested
.egg directories, and an .egg directory may contain arbitrarily
nested .egg directories or .egg zipfiles.
However, if you're not implementing dependency resolution, you don't
need that support. If such a nested item were in use, it'd be on
sys.path already. (In other words, the nesting lookups are only
useful for alternate version discovery, not simple metadata lookups.)
I'd like to bundle some third party eggs with my application without
installing them. Typically, I'd read __file__ and alter sys.path at
runtime to include the third party packages, but I understand that will
not work with zipped eggs.
I don't know the best way to go about this, but I at least want to know
a clean way to do it without requiring the third party eggs to be
downloaded and installed separately. Consider this layout.
theapp/
docs/
setup.py
theapp/
__init__.py
app.py
ext/
thirdparty1.egg
thirdparty2.egg
thirdparty3.egg
tests/
From my current understanding about how setuptools works, I think I
could treat the third party eggs as data files and and runtime
(somehow?) insert them into the path and import them.
I'm trying to work from this example.
http://peak.telecommunity.com/DevCenter/PythonEggs#accessing-package-resour…
Is this a workable approach? Is there a better way?
--Randall
At 11:10 PM 5/14/2009 +0200, Tarek Ziadé wrote:
>2009/5/14 P.J. Eby <pje(a)telecommunity.com>:
> > IOW, this approach keeps simple things simple, and leaves complex things
> > possible. It also does less I/O than what you're proposing, since in the
> > normal case the directories are only ever searched once, and the actual
> > metadata reads are both lazy and cached.
>
>Makes a lot of sense yes. I think I'll just start a prototype for that
>code, propose 376 on python-dev and see.
Notice, by the way, that this is a good example of pkg_resources'
code only looking complex if you haven't gone through all the
requirements and design issues yourself. The further you go with
this, the more likely you're going to end up with something very much
resembling pkg_resources itself.
The bad news is that you'll be reinventing a lot of wheels. The good
news is, you should also be able to understand and maintain
pkg_resources internals by the time you're done. ;-)
The relevant classes in pkg_resources, btw, are WorkingSet,
Distribution, and EntryPoint. You might want to just copy them and
strip out the parts you don't need. (i.e. sys.path manipulation and
requirements-resolving stuff)
At 12:03 PM 5/14/2009 +0200, Tarek Ziadé wrote:
>Hello
>
>for PEP 376, I have one last fuzzy point.
>
>http://svn.python.org/view/peps/trunk/pep-0376.txt?view=markup
>
>The "get_egg_info" api is currently based on scanning the whole
>sys.path. And since sys.path can be modified by people,
>so the algorithm is linear and can slow down when there are a lot of paths.
>
>I have a proposal: let's restrict the search for this API to
>site-package directories only. (directories added with
>site.addsitedir)
>
>People will be able to mark add any directory (like the per-user
>site-package directory - http://www.python.org/dev/peps/pep-0370)
>
>This requires to add in site.py a registry to keep track of all
>directories added through site.addsitedir
>
>Any thoughts ?
What tradeoffs are you optimizing for? Note that a single scan of
every directory on sys.path is exactly what happens when an import
doesn't find its target until the *last* directory on sys.path. So
this is not really a big deal if you're only doing it *once*.
If you want to optimize for repeated searches, the best way to do
this is with a structure like pkg_resources' WorkingSet object - it
simply reads the directories once and makes an object for each
installed package. These objects don't do any further I/O, so really
we're just talking about caching a list of .egg-info filenames.
Each object in the set can be queried for its metadata -- in which
case it reads it exactly once, and caches it.
With this setup, the full directory scan is only ever done once --
and it's basically equivalent to adding an extra import at the time
you first import the metadata management module.
Yes, it does mean a global, unless you want to hand off cache
management to the application. But the way pkg_resources does it,
with WorkingSet and Distribution objects, allows an app with special
needs to do its own path management and search operations.
IOW, this approach keeps simple things simple, and leaves complex
things possible. It also does less I/O than what you're proposing,
since in the normal case the directories are only ever searched once,
and the actual metadata reads are both lazy and cached.
Note, too, that site-packages dirs are likely to have more packages
on them than other directories, which means you're not necessarily
saving much I/O to start with, and even that small savings evaporates
as soon as you do more than one lookup for plugins.
At 10:19 AM 5/14/2009 +0200, Tarek Ziadé wrote:
>from a setuptools user point of view, the benefit I can see is that
>they will have better
>version numbers
Better how? That was my question. I personally find the common
version patterns in use (e.g. '-' and '-r' for post-releases) less
clunky and easier to read than 'post'. I also don't care for '.' in
front of dev and post. So, I don't see the changes as all "better"
from a readability POV.
That's not to say it's a bad thing, if it gives other benefits. But
it has not been stated what the benefits are supposed to be.
>that will comply with a documented standard, that is
This isn't actually a change; setuptools also has a documented standard.
>usable and understandable by os packagers for example.
I agree that a canonical form is useful. However, if that canonical
form can be generated from their existing version numbers, then why
should they bother changing?
That's the open question, IOW.
>Now I suppose setuptools will have to propose both for some time,
I don't see why. AFAICT, RationalVersion is a strict subset of the
syntax supported by setuptools, and most setuptools versions in use
should be convertible.
>But what I am scared of is : who will work on setuptools side ? can
>you bless someone to do the
>work when we agree on a common roadmap ?
I don't see that there is any work *to* do in setuptools core, since
RationalVersion is a strict subset, and we have setup-argument
validation providers already. I.e., anyone can make a plugin that
validates or converts a setup(version="...") string, put it on
sys.path, and force canonical versions.
