I want to ask you a question regarding something I want to do with my app.
The reason I'm asking here is because I understand that Distribute somehow
installs itself as "setuptools". (I've never looked into the details of it.)
And I want something similar, so maybe you can advise me how to do it.
I'm developing a package called `garlicsim`. The package is intended for
Python 2.X, but I am also offerring Python 3 support on a different fork
So both of these packages live side by side on PyPI, and Python 3 users
install `garlicsim_py3`, and Python 2 users install `garlicsim`.
The problem is: When third party modules want to use garlicsim, they should
have one package name to refer to, not two. Sure, they can do something like
import garlicsim_py3 as garlicsim
But I would prefer not to make the developers of these modules do this.
Is there a way that `garlicsim_py3` will install itself under the alias
`garlicsim`? What I want is for a Python 3 user to be able to `import
garlicsim` and refer to the module all the time as `garlicsim`, but that it
will really be `garlicsim_py3`.
(1) I've reached the decision to support Python 3 on a fork instead of in
the same code base; It's important for me that the code base will be clean,
and I would really not want to introduce compatibilty hacks.
2010/5/29 P.J. Eby <pje(a)telecommunity.com>:
> At 01:13 AM 5/29/2010 +0200, Tarek Ziadé wrote:
>> The problem I have with this approach is that we need to manage
>> somewhere at PyPI a list of potential installers,
>> and maybe deal with upgrades and replacements. Plus, I am not sure
>> that a user will really understand what to
>> do when he's asked to chose an installer. Sounds like something we
>> should only ask to power users, and
>> people that know what they are doing with p7g. So a bootstrap script
>> is useless for them.
> Actually, the way it would (presumably) work would be just a script like
> Guido's that downloads a source archive from PyPI and runs setup.py install.
> So, it's not really a matter of "choosing" an installer -- it'd just be,
> "download and install a package from source".
> If the package itself needs one of those other things in order to build
> itself or install dependencies, then the next stage of bootstrap is its
> problem (e.g. via ez_setup), not the stdlib's.
> So, I don't see the "power users only" objection making sense, at least if
> we drop back to what I believe was Guido's original proposal for a bootstrap
> script a few years ago.
I was not thinking about this proposal. If this what Guido proposed at
the summit, then I misunderstood. I was thiking about a bootstrap
process on the end-user side, to set up an installer once for all, on
a fresh Python.
The problem with the feature you describe (bootstrap embed in the
archive itself) is that we imply that the packager chooses himself an
installer and forces the end-user to use it when he installs the
software. This means that the end user might end up with several
installers installed for the same Python. It also implies that the
developer provides a smart setup.py script that embeds that bootstrap,
and runs some code.
I think separating the concerns and letting the end user pick/use
explicitly *one* installer globally is better because several
installers won't compete on the target system (even if we supposely
want them all to be compatible in the future). Of course, this is only
true for source distributions.
It will also allow distributions to be "dumb" envelopes with static
metadata that are the same all the time, no matter which tool created
them, and eventually remove setup.py in favor of statically described
metadata using PEP 345.
Today, for instance, if an installer wants to install a distribution
based on setuptools, it has to run the "egg_info" command to extract
the metadata, on the target system. Being able to get those metadata
without running any code would be better. For instance, installers
could list all dependencies for a project by querying PyPI with zero
download/execution. (thanks to PEP 345 environment markers)
What would make more sense I think, would be to have all installers an
identical archive for a given project, that doesn't need more code to
be run to get all the metadata.
So at the end, the end user would chose an installer that is
compatible with these archive, and know how to install them. In other
words, have ez_setup for example, run once for all at the Python
level, and be THE installer. Or run a pip_setup or whatever.
Tarek Ziadé | http://ziade.org
At 01:11 AM 5/28/2010 +0200, cool-RR wrote:
>The thing I want the most right now is anyone's opinion of what
>could possibly go wrong with the scheme that P.J. offered, before I
>start moving my codebase to it.
It's quite routine. Certainly, few of my PyPI projects have the same
project and package name. For example, 'Importing' contains the
It's also done by many other packages I'm not involved with. For
example, 'AeroCalc' is the PyPI name of a project that ships an
'aerocalc' package, 'AdjectorClient' ships various packages such as
adjector.core and adjector.model... and those are just two things I
found by randomly clicking on the A's in the giant list of PyPI packages!
I'm sure there will be some found in every letter of the alphabet. ;-)
The only thing that can go wrong is if somebody installs the Python 2
and Python 3 versions to the same installation location. However, as
long as both versions alert the user when used on the wrong Python
version, they will quickly discover their mistake. ;-)
At 10:19 AM 5/29/2010 +0200, Tarek Ziadé wrote:
>I was not thinking about this proposal. If this what Guido proposed at
>the summit, then I misunderstood.
I don't know what he proposed at the summit - I'm referring to the
bootstrap script he wrote that actually does this. It was a couple
years ago on Python-Dev, IIRC.
>The problem with the feature you describe (bootstrap embed in the
>archive itself) is that we imply that the packager chooses himself an
>installer and forces the end-user to use it when he installs the
>software. This means that the end user might end up with several
>installers installed for the same Python.
So? It's not like they're going to accidentally run 'easy_install'
when they meant to type 'pip', or vice versa. ;-)
>It also implies that the
>developer provides a smart setup.py script that embeds that bootstrap,
>and runs some code.
Only if they need something beyond what the distutils provide.
>It will also allow distributions to be "dumb" envelopes with static
>metadata that are the same all the time, no matter which tool created
>them, and eventually remove setup.py in favor of statically described
>metadata using PEP 345.
For packages with complex build requirements or distutils extensions
(e.g. numpy), this is unlikely to happen any time soon.
Conversely, for packages where this *is* the case, the current
distutils is adequate, and having a bootstrapper that can install
them would be a win.
>Today, for instance, if an installer wants to install a distribution
>based on setuptools, it has to run the "egg_info" command to extract
>the metadata, on the target system.
It's a good thing Guido loaned me the time machine so I could go back
to 2005 and fix that one, then:
Five years should be long enough ago that by now all the
setuptools-based source distributions on PyPI today will already have
the metadata you need... so when you go back and try again to build
your new installer today, it will already be there for you to
use. Just download any random setuptools-based sdist from PyPI and
run "tar -tzf" or "unzip -v" on it to see!
For that matter, it's long enough ago that you should also have the
fix already retroactively included in Distribute as well, so you
should probably even be able to look at one of Distribute's own
source distributions and see that it now contains the metadata you
I've been working with svn checkouts of a couple of packages by placing
them in the develop key of my default.cfg.
Now, I want to stop doing that, so I removed the lines from default.cfg
and re-ran the buildout.
I was expecting to have complaints about packages not being available,
but instead, the develop eggs (ie: the svn checkouts) just kept on being
...I ever tried putting in specific requirements in the versions section
of my buildout, that I knew the develop eggs wouldn't meet (r100000
wont' exist for a while yet!)
No dice, the develop eggs in the checkouts still get used!
Here's selected output:
$ bin/buildout -vv
repoze.bfg.chameleon_genshi = 0.1-r9432
repoze.bfg.xmlrpc = 0.3
We have a develop egg: demo release
Getting required 'repoze.bfg.chameleon-genshi'
required by demo release.
We have a develop egg: repoze.bfg.chameleon-genshi 0.1-r9431
Getting required 'repoze.bfg.xmlrpc==0.3'
We have a develop egg: repoze.bfg.xmlrpc 0.3
How do I get buildout to stop using these develop'ed checkouts and go
get a package from an index?
We have a collection of private eggs served from a simple folder served
up by Apache.
The following works fine to get these eggs via buildout:
find-links = http://username:password@server/eggs-index/
However, the following does not find the eggs:
However, if I mount up the folder served by apache and do:
bin/easy_install --find-links=/mnt/path/to/eggs-index/ eggname
...it works fine.
...bombs out complaining that authorization is required.
So, how come the behaviour of all of these is different?
Distutils2 is going to be added back in Python (hopefully in 3.2) and
without an install script, it's pretty useless as-is.
We've discussed during the summit at Pycon to create some kind of
bootstrap script in Python, to allow people to
set up an installer of their choice, but I think it's a bad idea.
= summary of the summit proposal =
- an "install" script is added in the scripts, for people to install
distributions in Python. Could be called "pyinstall" for example.
- when first used, it would ask the user to choose a third-party
installer (like Pip). Then it would download it and
install it with a simple "python setup.py install"
- from there, the install script would be linked to that installer.
If I recall it correctly, this feature was proposed to be able to have
a "modern" installer in Python without
including it in the standard library. (so it can have its own shorter
release cycles). The bootstrap story
would just make it easier for people to get an installer, without
having to do extra manual steps.
The problem I have with this approach is that we need to manage
somewhere at PyPI a list of potential installers,
and maybe deal with upgrades and replacements. Plus, I am not sure
that a user will really understand what to
do when he's asked to chose an installer. Sounds like something we
should only ask to power users, and
people that know what they are doing with p7g. So a bootstrap script
is useless for them.
= alternative proposal =
Let's add that script but powered by Distutils2. It could be Pip if
people from this project think it's a good idea and want to merge, or
an easy_install derivation, or a new script from scratch.
IOW: you get an installer for free in the stdlib without having to think.
Now for the problem of the release cycle (e.g. once in the standard
library it has to wait 18 months for a new version),
I propose that Distutils2 allow the usage of a third party installer
through configuration. IOW, Distutils2 would ship with an installer,
but could use through a simple change in distutils.cfg, another one
installed by a third party project that is more recent.
For this to work, we can define an installer standard interface ala
wsgi. Basically, we state that an installer has to implement a simple
function that takes a name of a project to install, and an optional
def install(name, version=None): # if version is None, it means
the latest one.
This needs more work, uninstall is missing in that description, and
what about the script options, etc. but you get the idea: make sure
people can use the installer of their choice, if it turns out that the
one provided by Distutils2 is not good enough anymore for any reason.
Any opinion ?
Tarek Ziadé | http://ziade.org