At 01:56 PM 6/19/2007 -0500, Edward Ream wrote:

Many thanks, Phillip, for your quick and generous response.

...

Eggs are a binary distribution format, originally developed to support user-installed plugins for systems like Chandler, Zope, etc. They aren't a source distribution format; sdist works well enough for that and for easy_install if you have a pure-Python package (or your users have C compilers).

Oh joy. sdist is working fine--Leo is indeed a pure python program.

...

I'd suggest using the --show-response option so you can see what's happening.

Will do, assuming there is still a problem with uploading the .zip file.

...

here's the output from my attempt at downloading from the SF page:

$ easy_install -nvf http://sourceforge.net/project/showfiles.php?group_id=3458 leo

Excellent. I'll try again with the options you show.

Note that this means you can use the above URL as your "Download URL" for PyPI; I just specified it using -f because that's not what you currently have up as the link on PyPI. The -n is because I didn't want to *actually* install leo, and the -v was for verbosity. If you update your "Download URL" you should be able to simply "easy_install leo" and have it work.

At 09:43 AM 6/20/2007 -0500, Edward Ream wrote:

...

...

http://sourceforge.net/project/showfiles.php?group_id=3458 leo ...you can use the above URL as your "Download URL"

Success, or close to it. Here is the present status.

1. setup.py register works.

2. setup.py sdist upload --show-response fails as follows:

[big snip: creating files] removing 'leo-4.4.3preview10' (and everything under it) running upload Submitting dist\leo-4.4.3preview10.zip to http://www.python.org/pypi Upload failed (400): Bad Request --------------------------------------------------------------------------- Error processing form invalid distribution file

Not a big deal. Any idea what might be happening? (see the P.S. for the call to setup.py)

No. I'd suggest following up with the Catalog-SIG or the PyPI support tracker.

3. After uploading 'leo-4.4.3preview10.zip to SourceForge, easy_install leo -v works (I think), but I was surprised that easy_install builds an egg:

[big snip: copying, compiling and building the egg] Installed c:\python25\lib\site-packages\leo-4.4.3preview10-py2.5.egg Processing dependencies for leo Finished processing dependencies for leo

I'm also a surprised that this new egg did not become the default version of Leo.

It did; your setup() is broken. See below.

setuptools.setup ( name='leo', version='4.4.3preview10', # No spaces! # pre-release tags: 4.4.3b1 or 4.4.3rc1 or 4.4.3preview1 # Do not use post-release-tags: 4.4.3-whatever. # final release: 4.4.3final or just 4.4.3. author='Edward K. Ream', author_email='edreamleo@charter.net', url='http://webpages.charter.net/edreamleo/front.html', download_url='http://sourceforge.net/project/showfiles.php?group_id=3458', #'http://downloads.sourceforge.net/leo/leo-4.4.3preview9.egg',

#'http://sourceforge.net/project/showfiles.php?group_id=3458&package_id=29106',

# py_modules=[], # The manifest specifies everything.

Sorry, you can't not list the modules and still install correctly or build correct binary distributions. py_modules isn't optional. That's as true for setuptools as it is for distutils. However, it's not clear to me whether you're intending to distribute a bunch of modules under 'src', or a package called 'leo'... Your sitecustomize implies that you have a bunch of modules under 'src' -- in which case you should list them in py_modules, and include a "package_dir={'': 'src'}" along with a bunch of other tricky-to-get-right things. However, the next line of your setup() implies that you intend to install some packages instead...

packages = setuptools.find_packages(),

This line will find a bunch of packages named "plugins", "scripts", "src", "modes", "Icons", "config", "dist", "doc", "test", "www", "extensions", "extensions.Pmw", etc. This looks completely hosed, because I assume these should all be subpackages of "leo", not top-level packages. And you probably don't want a package called "src"! So, your layout and setup are quite hosed, regardless of whether you're using distutils or setuptools. What is it that you intend to distribute? Is it one package called 'leo' with a bunch of subpackages? Or what? Is everything in there really a package? (e.g. why is 'doc' a package?)

exclude_package_data = { '': ['*.pyc','*.pyo']},

This line is unnecessary; setuptools doesn't consider those files to be package data anyway, and it won't stop them from being put into any eggs built by easy_install.

At 01:35 PM 6/20/2007 -0500, Edward Ream wrote:

...

Sorry, you can't not list the modules and still install correctly or build correct binary distributions.

Ok. I'm using sdist, not bdist_x. I assume that doesn't matter?

It matters if you want someone to be able to *install* your package. Most bdist_* commands are based on "build" and "install", which both need to know the modules, if you're installing any standalone modules. They also need to know about the packages, but it appears that you don't actually *have* any packages.

...

What is it that you intend to distribute? Is it one package called 'leo' with a bunch of subpackages? Or what? Is everything in there really a package? (e.g. why is 'doc' a package?)

It seems my cluelessness has managed to confuse even you. As you no doubt have guessed, this is not easy for me.

I'm not sure of the terminology, so please be patient. What I want to do is distribute Leo (a pure Python application, packaged as a single entity--a package?) ,

A packaged entity is a "project" - the physical distribution of that entity is a "distribution". A "package" is a directory with an __init__.py and zero or more other .py files. Currently, for example, you have a package called 'src', containing various modules. If you had tried "from src import leo", or "import src.leo", for example, your attempt at using your easy_install-ed leo would have worked. (Of course, everything after that point would likely have failed.)

consisting of *all* the files under cvs management in leo directory and subdirectories of the leo directory that contain a cvs folder.

The distutils and setuptools do not install "files" - they install Python modules, scripts, extensions, and packages. It appears to me as though you do not have a project that can be meaningfully installed or distributed using the distutils or setuptools, as you have a collection of directories containing files, along with some directories with modules and scripts. In other words, more of an "application" than a "distribution". The distutils and setuptools can only support "applications" that are composed solely of Python modules, scripts, extensions, and packages (including *read-only* data files inside the packages). They can't install arbitrary trees of files, which unfortunately is what you have. You would need to reorganize your code in such a way that it does not contain any directories that you expect the user to modify, for example. Sample configuration files can still be included as package data, but your program would have to explicitly read them and copy them to a runtime configuration location. To put it another way, disutils and setuptools are designed to install *immutable* collections of code and data. At runtime, your program can read from the immutable code or data, but any writable runtime data must be placed somewhere else. Otherwise, you can't install a single copy of the project for use by all users on a system, nor is there any way to safely uninstall or upgrade it.

...

Your sitecustomize implies that you have a bunch of modules under 'src' -- in which case you should list them in py_modules, and include a "package_dir={'': 'src'}" along with a bunch of other tricky-to-get-right things.

Ok. I can do that. Are you implying that this is a less-than-recommended approach?

Yes.

At 07:33 AM 6/21/2007 -0500, Edward Ream wrote:

...

It appears to me as though you do not have a project that can be meaningfully installed or distributed using the distutils or setuptools,

Doesn't this say more about distutils and setuptools than about Leo?

Depends on what you mean by "say". :) Per http://www.python.org/doc/2.1/dist/ , the distutils were created "to make Python modules and extensions easily available to a wider audience with very little overhead for build/release/install mechanics." They aren't for installing arbitrary applications. It just happens that, if your "application" consists solely of scripts that tag along with some libraries and immutable data, then they also work. It's not reasonable to expect the distutils to be a general purpose software installation toolkit; they are, after all, for sharing Python libraries.

...

...

To put it another way, disutils and setuptools are designed to install > *immutable* collections of code and data.

The distinction between mutable and immutable data makes some (but not complete) sense on Linux. It is irrelevant on Windows.

Actually, Microsoft has been trying to get people to stop putting mutable data in the \Program Files tree for many years now. That's why user-specific $APPDATA directories exist there.

Here is what I particularly like about setuptools:

setup.py sdist register upload easy_install leo cleverly uses a generic upload_url setuptools.set can (at least sometimes) find all files managed by cvs.

Why do any of the above require any particular organization of files?

They don't -- but a LOT of other features of easy_install *do* depend on them. Notably, easy_install is intended to support (relatively) clean uninstallation, and the simultaneous installation of multiple versions of a project along the same PYTHONPATH. Aside from locating files in the first place, most of easy_install's complexity comes from either 1) supporting near-arbitrary distutils-based projects or 2) managing sys.path. Your project doesn't use either of these features, but they're absolutely vital for setuptools' *real* goal, which is to make widespread library sharing and reuse practical. The distutils do fine for distributing individual projects, but does badly for inter-project dependencies. This makes people who want to depend on other libraries either bundle them inside their project, or write their own replacement just to avoid the hassle of updating their bundled versions of things. Then, you either end up with multiple copies of some library installed anyway, or version skew/stomp. Setuptools was intended to fix this by making it easy to say, "my library uses these other libraries" -- and have them automatically installed, with version skew problems handled. This is a delicate enough problem to solve without throwing mutable files into the mix.

We in the Leo community never argue over user preferences because my guiding design principle is this: if there are two or more even halfway reasonable ways of doing something, Leo provides an option that allows users to choose what works for them, and that's is the end of the discusssion. It seems that allowing me to distribute Leo as I and Leo's users prefer would be a useful addition to setuptools.

Projects that don't consist of packages, immutable data, and scripts, don't really support library reuse very well. In fact, projects that include mutable files, would be in direct *opposition* to easy_install's goals for package management, so it really shouldn't be surprising that it doesn't support them very well. :) If you'd like to use the distutils or setuptools to distribute leo, I would suggest restructuring it as a set of Python packages rather than as a collection of modules and directories. (E.g., "import leo.config" rather than "import leoConfig".) For plugin support, I would suggest you investigate setuptools "entry points" system, which many packages and applications use to automatically discover separately-installed plugins: http://peak.telecommunity.com/DevCenter/setuptools#extensible-applications-a... http://peak.telecommunity.com/DevCenter/PkgResources#entry-points This method has the advantage that people can distribute plugins as setuptools-based projects, and users can install them with "easy_install pluginprojectname". (See, for example, all the various TurboGears widget plugin projects on the Cheeseshop, the TG/Buffet templating engines, Chandler plugins, etc., for examples of plugin systems built this way.) I'm not saying this would be easy or simple; many existing projects with other plugin or extension systems or using other distribution mechanisms have taken some time to migrate or are still migrating to this approach, in order to take advantage of the benefits. And truthfully, most projects using the full spectrum of setuptools' plugin abilities are relatively new, written after the features were available. Certainly, you could also retain older plugin mechanisms, and scan configured directories for plugins in addition to any "built-in" plugins that you provide in your main distribution project, so that you keep continuity for your existing users. Of course, you may also decide that the cost/benefit isn't there for you, and that's understandable. In that case, you may wish to use some other distribution mechanism, or create your own; setuptools unfortunately can't both achieve its own goals, and support installing non-importable things (aside from scripts and immutable package data) or arbitrary installation locations for project contents.

Does anyone know of documentation for creating debian packags for pure Python projects? In particular, I would like to bypass the make-centric build process and the debian/rules file. Googling has turned up pages like http://blog.mypapit.net/2006/02/create-you-own-debianubuntu-deb-package.html which contains other links, but I am wondering whether more Python-specific pages exist. Yes, this is slightly OT, but I'm thinking that creating a debian package for Leo might lead me in a direction more compatible with distutils and easy_install. Thanks! Edward -------------------------------------------------------------------- Edward K. Ream email: edreamleo@charter.net Leo: http://webpages.charter.net/edreamleo/front.html --------------------------------------------------------------------

Edward Ream wrote:

Does anyone know of documentation for creating debian packags for pure Python projects? In particular, I would like to bypass the make-centric build process and the debian/rules file.

The official Debian Python Policy document is the best starting point. http://www.debian.org/doc/packaging-manuals/python-policy/ The Wiki seems to have some more information, too. http://wiki.debian.org/DebianPython/NewPolicy You cannot really get around debian/rules, though. The debian-python mailing list would be the place to ask further questions along this line. http://lists.debian.org/debian-python/ -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco

At 10:12 AM 7/7/2007 -0600, Zooko O'Whielacronx wrote:

On Jun 19, 2007, at 10:33 AM, Phillip J. Eby wrote:

...

...

2. The resulting egg file is much larger than the .zip file created by 'python setup.py sdist' in spite of exclude_package_data = { '': ['*.pyc','*.pyo']},

Is there any way to exclude .pyc and .pyo files?

No; you can exclude the source if you like, though, with --exclude-source-files. Eggs are a binary distribution format, originally developed to support user-installed plugins for systems like Chandler, Zope, etc. They aren't a source distribution format; sdist works well enough for that and for easy_install if you have a pure-Python package (or your users have C compilers).

I would prefer to ship compiled C modules and source .py files. For one thing, compressed source .py files are usually around 25% smaller than compressed .pyc files. For another thing, I value making source available to the end-user, and I am loathe to ship non-source unless I really have to (which I pretty much do for C extension modules).

So, please accept this as a feature request to make the above- mentioned behavior configurable for users such as me.

For the reasons I already explained above, this is a no-go. If you want to ship binaries plus source, ship eggs. If you want to ship without .pyc/.pyo files, ship an sdist. This distinction is not about your preferences as a distributor, but about your users. As a user, I don't want to pay for .py-compilation every time I import your modules, just to save you a few extra bytes of bandwidth. If the option existed to ship without compiled modules, people would use it. Following which, other people would begin claiming that "eggs are slow". And all this, for what actual benefit? Absolutely zip! ...no pun intended.