distutils (packaging) newbie questions please....

[Alex Martelli]

Christian Seberino wrote:

> I really want to learn distutils to professionally package some open
> source software.
> 
> I had some distutils questions after reading python.org distutil docs
> on packaging code
> if you don't mind.  I would really appreciate some help.....
> 
> 
> 0. I'm confused about use of the word "package" in setup.py.
>     packages = ["<package1>", "<package2>", ...]
>     Does package here just mean "a new directory of source code to
> consider"???

No, it means a Python package.  The concept of "package" (a module
that may contain other modules, and corresponds to a directory with
an __init__.py file) exists in Python quite independently from
distutils -- the distutils just SUPPORT the concept, if the code
you want to package-up uses Python packages.


> 1. What exactly goes into the __init__.py files? I know what to put in
> setup.py
>     and I know __init__.py is used for packages but I don't know what
> to put IN this file.

Again, not a distutils-related question.  A Python package's __init__.py
has the module-body of the package (remember, any package IS a module).
If you need no Python statements whatsoever in that body, __init__.py
may be empty (it must still exist, or else Python will refuse to
consider that directory as a package).  Normally you do want, at least,
to assign the appropriate list to __all__ .

In p. 124 of "Python in a Nutshell", i.e. the section named "Packages",
I summarize the issue as follows:
"""
A package named P resides in a subdirectory, also called P, of some
directory in sys.path.  The module body of P is in the file P/__init__.py.
You must have a file named P/__init__.py, even if it's empty (representing
an empty module body), in order to indicate to Python that directory P
is indeed a package.
"""

Do you find this unclear, or was it hard for you to find -- or didn't
you even considering checking out the Nutshell book before setting
yourself the task to "professionally package" some Python software?
Feedback on this (private or public as you wish) may help me do a
better job next time.  [Note that if the issue is with the purchase
price of the Nutshell book, there IS a perfectly legal way to read it
for free: the "Safari" online book service of O'Reilly offers the
first two weeks free, so you can join it, and read Python in a Nutshell,
which is among the over 1000 books on Safari, paying nothing as long
as you cancel before two weeks are up].


> 2. About the data_files option for setup.py....
> 
>       data_files = [ "<installation directory>", [ "<file1>",
> "<file2>", ...] ]
> 
>    Is there a way to AVOID having to type EVERY single data file?? Can
> I specify just the directory or use wild cards??

You can form the list of file names in any way you wish, including,
for example, using the glob module from Python's standard library.
setup.py is a Python script, and any Python construct is available
to you as the author of that script.

For example, suppose we wanted to package up (for installation under
directory foodir) all the files in our directory named datadir EXCEPT
those whose file-extension is '.not' .  That's easy...:

"""
from distutils.core import setup

import glob

thefiles = []
for afile in glob.glob('datadir/*'):
    if afile.endswith('.not'): continue
    thefiles.append(afile)

print "Packaging up", thefiles

setup(name='justdata', data_files = [ [ 'foodir', thefiles ] ] )
"""

Here's an example of how this works:

[alex at lancelot tryp]$ ls datadir
aa.yes  bb.not  cc.yes
[alex at lancelot tryp]$ python setup.py bdist
Packaging up ['datadir/aa.yes', 'datadir/cc.yes']
running bdist
running bdist_dumb
running build
installing to build/bdist.linux-i686/dumb
running install
running install_data
creating build/bdist.linux-i686/dumb
creating build/bdist.linux-i686/dumb/usr
creating build/bdist.linux-i686/dumb/usr/local
creating build/bdist.linux-i686/dumb/usr/local/foodir
copying datadir/aa.yes -> build/bdist.linux-i686/dumb/usr/local/foodir
copying datadir/cc.yes -> build/bdist.linux-i686/dumb/usr/local/foodir
tar -cf /x/tryp/dist/justdata-0.0.0.linux-i686.tar .
gzip -f9 /x/tryp/dist/justdata-0.0.0.linux-i686.tar
removing 'build/bdist.linux-i686/dumb' (and everything under it)
[alex at lancelot tryp]$ tar tvzf dist/justdata-0.0.0.linux-i686.tar.gz
drwxrwxr-x alex/alex         0 2003-10-01 14:25:40 ./
drwxrwxr-x alex/alex         0 2003-10-01 14:25:40 ./usr/
drwxrwxr-x alex/alex         0 2003-10-01 14:25:40 ./usr/local/
drwxrwxr-x alex/alex         0 2003-10-01 14:25:40 ./usr/local/foodir/
-rw-rw-r-- alex/alex         0 2003-10-01 14:13:48 ./usr/local/foodir/aa.yes
-rw-rw-r-- alex/alex         0 2003-10-01 14:13:55 ./usr/local/foodir/cc.yes
[alex at lancelot tryp]$


> 3.  python setup.py sdist   will make a package of SOURCE code and
>     using "bdist" will put BINARIES in package.  What does this
> mean???
>     Will bdist packages unpack to pyc (bytecode) files and sdist will
> unpack
>     py (noncompiled) files????  What if I wanted py and pyc files
> installed
>     from the package?? (i.e. source and compiled versions)

Hmmm, if the issue was so unclear (and important) to you, was it
SO hard to try things out...?  Say that setup.py is:

from distutils.core import setup

setup(name='onemodule', py_modules=['hello'])

and hello.py in the same directory is:

def greet(name='World'):
    return 'Hello, %s!' % name

Now, running python setup.py bdist gives the output:

[alex at lancelot tryp]$ python setup.py bdist
running bdist
running bdist_dumb
running build
running build_py
creating build/lib
copying hello.py -> build/lib
installing to build/bdist.linux-i686/dumb
running install
running install_lib
creating build/bdist.linux-i686/dumb
creating build/bdist.linux-i686/dumb/usr
creating build/bdist.linux-i686/dumb/usr/local
creating build/bdist.linux-i686/dumb/usr/local/lib
creating build/bdist.linux-i686/dumb/usr/local/lib/python2.3
creating build/bdist.linux-i686/dumb/usr/local/lib/python2.3/site-packages
copying build/lib/hello.py ->
build/bdist.linux-i686/dumb/usr/local/lib/python2.3/site-packages
byte-compiling
build/bdist.linux-i686/dumb/usr/local/lib/python2.3/site-packages/hello.py
to hello.pyc
tar -cf /x/tryp/dist/onemodule-0.0.0.linux-i686.tar .
gzip -f9 /x/tryp/dist/onemodule-0.0.0.linux-i686.tar
removing 'build/bdist.linux-i686/dumb' (and everything under it)

Notice that hello.py is copied into the directory to be packaged, as
well as hello,pyc.  And indeed we can double check what's in the tarball:

[alex at lancelot tryp]$ tar tvzf dist/onemodule-0.0.0.linux-i686.tar.gz
drwxrwxr-x alex/alex         0 2003-10-01 14:30:41 ./
drwxrwxr-x alex/alex         0 2003-10-01 14:30:41 ./usr/
drwxrwxr-x alex/alex         0 2003-10-01 14:30:41 ./usr/local/
drwxrwxr-x alex/alex         0 2003-10-01 14:30:41 ./usr/local/lib/
drwxrwxr-x alex/alex         0 2003-10-01 14:30:41
./usr/local/lib/python2.3/
drwxrwxr-x alex/alex         0 2003-10-01 14:30:42
./usr/local/lib/python2.3/site-packages/
-rw-rw-r-- alex/alex        56 2003-10-01 14:30:10
./usr/local/lib/python2.3/site-packages/hello.py
-rw-rw-r-- alex/alex       335 2003-10-01 14:30:42
./usr/local/lib/python2.3/site-packages/hello.pyc
[alex at lancelot tryp]$

See?  Both the .py AND the .pyc, as expected.

Should you want to REMOVE the .py files, for example, that can
easily be achieved by operating directly on the tarball or zipfile,
after distutils is finished (customizing distutils for such
purposes, on the other hand, is harder).


Alex