Gerry Reno wrote:

Ok, I got the __file__ problem solved but now I want to do this whole deb pkg create as just a single command in my own setup.py. How can I do this?

I'm thinking something like:

# my setup.py import stdeb

Command(mycmd): initialize_option... finalize_option... run sub_commands=[('sdist_dsc', None),] system('dpgk-source ...) print ".deb is here..."

Would this work and is this how to do it? Do you really need to run it directly from setup.py? Can't you run it from a shell script or write a python script making lots of use of subprocess? The reason I ask is that distutils isn't fun to extend, and as you're already going to be running on a Debian-based system, I can't see what's to be gained.

What you want to do is probably possible, but I don't have the motivation to do it myself. I guess it could allow a bdist_deb option, but IMO that's not particularly desirable -- the Debian source package emitted by stdeb can be compiled for any Debian derivative for any version (provided the dependencies are met) -- whereas the .deb will only have defined behavior on the Debian/Ubuntu release on which it was built. -Andrew

Gerry Reno wrote:

...

What you want to do is probably possible, but I don't have the motivation to do it myself. I guess it could allow a bdist_deb option, but IMO that's not particularly desirable -- the Debian source package emitted by stdeb can be compiled for any Debian derivative for any version (provided the dependencies are met) Having a single command for .deb doesn't preclude stdeb from emitting source package .dsc.

OK, if you're sure... :) I'd suggest attempting to add a "bdist_deb" command to stdeb's stdeb/command/ directory for use with distutils. Then register it with entry_points in the setup.py file. Note that you wouldn't want bdist_deb to inherit the binary generation stuff from bdist_dumb or friends -- you'd specifically want to call out the the appropriate debian command (dpkg-buildpackage). I'd be happy to include such a thing with stdeb, though I'd probably not use it much myself -- I like having a single source package that I just recompile for the different distributions. -Andrew

Ok, I have a setup.py that imports stdeb and creates a 'bdist_deb' command. The entire thing is working except for the last subcommand for dpkg-buildpackage is trying to build for python2.4 and failing. dpkg-buildpackage fails in the fakeroot... ... debian/rules build python2.4 -c "import setuptools,sys;f='setup.py';sys.argv[0]=f;execfile(f,{'__file__':f,'__name__':'__main__'})" build Traceback (most recent call last): File "<string>", line 1, in ? File "setup.py", line 35, in ? import stdeb ImportError: No module named stdeb make: *** [build-python2.4] Error 1 dpkg-buildpackage: failure: debian/rules build gave error exit status 2 Is this something to do with python versions? So how do I limit the python versions for dpkg-buildpackage? stdeb is 0.3 $ python -V Python 2.5.2 Regards, Gerry Gerry Reno wrote:

Ok, I got the __file__ problem solved but now I want to do this whole deb pkg create as just a single command in my own setup.py. How can I do this?

I'm thinking something like:

# my setup.py import stdeb

Command(mycmd): initialize_option... finalize_option... run sub_commands=[('sdist_dsc', None),] system('dpgk-source ...) print ".deb is here..."

Ok, I found how to control the python version: XS-Python-Version So now it is using python2.5 but dpkg-buildpackage hits an error: error: option --single-version-externally-managed not recognized What is this option? I've been searching but I don't find how to solve this. Is there a way to fix this? Regards, Gerry Gerry Reno wrote:

Ok, I have a setup.py that imports stdeb and creates a 'bdist_deb' command. The entire thing is working except for the last subcommand for dpkg-buildpackage is trying to build for python2.4 and failing.

dpkg-buildpackage fails in the fakeroot... ... debian/rules build python2.4 -c "import setuptools,sys;f='setup.py';sys.argv[0]=f;execfile(f,{'__file__':f,'__name__':'__main__'})" build Traceback (most recent call last): File "<string>", line 1, in ? File "setup.py", line 35, in ? import stdeb ImportError: No module named stdeb make: *** [build-python2.4] Error 1 dpkg-buildpackage: failure: debian/rules build gave error exit status 2

Is this something to do with python versions? So how do I limit the python versions for dpkg-buildpackage?

stdeb is 0.3 $ python -V Python 2.5.2

Regards, Gerry

Gerry Reno wrote:

...

Ok, I got the __file__ problem solved but now I want to do this whole deb pkg create as just a single command in my own setup.py. How can I do this?

I'm thinking something like:

# my setup.py import stdeb

Command(mycmd): initialize_option... finalize_option... run sub_commands=[('sdist_dsc', None),] system('dpgk-source ...) print ".deb is here..."

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

Gerry Reno wrote:

In my setup.py I have my own "install":

from distutils.command.install import install ... class myinstall(install): ...

and when we call dpkg-buildpackage this local "install" seems to maybe be causing the problem with the error: option --single-version-externally-managed not recognized.

How can we make this option be recognized? Do we need to inherit something from setuptools? I think there are 2 options:

1) upgrade to the nascent stdeb 0.4. It doesn't call setup.py with the --single-version-externally-managed argument. 2) inherit from setuptools' install command. I would suggest option 1. If you need me to make a release, I'm happy to do so -- but maybe that will be better after it has bdist_deb? :) -Andrew

Gerry Reno wrote:

Andrew Straw wrote:

...

Gerry Reno wrote:

...

In my setup.py I have my own "install":

from distutils.command.install import install ... class myinstall(install): ...

and when we call dpkg-buildpackage this local "install" seems to maybe be causing the problem with the error: option --single-version-externally-managed not recognized.

How can we make this option be recognized? Do we need to inherit something from setuptools?

I think there are 2 options:

1) upgrade to the nascent stdeb 0.4. It doesn't call setup.py with the --single-version-externally-managed argument.

I'm using Ubuntu 8.0.4 LTS Server and when I run apt-get it brings in stdeb 0.3. I always use the distro packagers to install software so how can I use apt-get to install stdeb 0.4?

stdeb isn't in the Ubuntu (or Debian) repository. And I thought you said you used easy_install on it. Are you sure you can't just upgrade by hand? I can't see how else you got it.

...

I would suggest option 1. If you need me to make a release, I'm happy to do so -- but maybe that will be better after it has bdist_deb? :)

My only concern now with stdeb adding the bdist_deb command is that it may not work like other 'bdist' commands. And having 'bdist_deb' work similar to 'bdist_rpm' is important. It would have to use setup.cfg for config and allow for pre/postinstall scripts to run.

Well, to produce the final .deb file, stdeb produces a source package and then, in another process, calls the debian machinery may be called to produce the .deb file (I guess you're now quite familiar with that). I feel this approach benefits from using the standard Debian infrastructure. Specifically, python-support and debhelper 7 called by stdeb 0.4 do things like: install a single copy of the .py files and symlink them into the per-version directories, compile .pyc files per-version at install time, compile extension modules per-version at build time, and so on. Reproducing that would be hard, error prone, and a pointless duplication of effort. And .debs that don't do all that stuff that will be sub-standard and won't behave like those downloaded from the Debian/Ubuntu repos. Thus, I really think calling the real Debian machinery is the way to go. As I understand it (which is not well), bdist_rpm calls out to an "rpm" command from the original "python setup.py bdist_rpm" process, but at that point the binary is already made at this is just a packaging step. With stdeb, the initial output will be the source package which would be compiled to a binary by another process -- a subprocess of dpkg-buildpackage. You can control this binary generation by modifying debian/rules, which is a Makefile that is run to build the binary. So you can make sure that building the binary as done by the debian/rules file handles the options in setup.cfg. It should do that automatically, since what happens is a glorified "python setup.py install". As far as pre/postinstall scripts -- what do you need them to do? It would be possible to add support for stdeb to run these scripts at debian .deb install time. (stdeb already runs a preinstall script to ease the python-central -> python-support transition. See commit 9faaf049.) But if they're limited in scope and merely modify the files you want installed, they're probably already run as part of the .deb build process if you've hooked into the "install" command. I can't see how an infrastructure based only on extending the distutils install command could be used to distinguish between A) "pre/post-install" scripts that merely modify the files to be installed and could thus be run at package build time versus B) pre/post-install scripts that actually need to be run once the .deb package is being installed on the target system that might do something like start a web server. It seems there has to be a specification of the difference. -Andrew

Gerry Reno wrote:

How do I get stdeb 0.4?

The easiest way: git clone git://github.com/astraw/stdeb.git cd stdeb sudo python setup.py install If you want a tarball, please look at my email 09/19/2009 09:36 PM -0700.

For bdist_rpm the way that you get pre and postpackage install scripts to run during package install is by using the following in the setup.cfg: [bdist_rpm] install_script = rpm_install_sh.txt

then rpm_install_sh.txt gets inserted into the rpm .spec file at the %install marker. Now you can also add other sections into rpm_install_sh.txt like %pre and %post which are sections of shell scripts that run prepackage and postpackage install. As far as 'python setup.py install' build type pre/postinstall, you can subclass install class and just run some scripts either side of parent "install" class. I think it would be straightforward to integrate something similar into stdeb.

I'm not too much in favor of piggybacking distutils' infrastructure to parse the setup.cfg file, though. Perhaps we could parse setup.cfg on our own (i.e. replace looking for stdeb.cfg with looking for setup.cfg and changing the section name appropriately). In the near term the easiest thing is to stick with the stdeb.cfg file for now and and a new field for each of pre/post install and pre/post remove. -Andrew

Gerry Reno wrote:

Andrew Straw wrote:

...

Gerry Reno wrote:

...

How do I get stdeb 0.4?

The easiest way:

git clone git://github.com/astraw/stdeb.git cd stdeb sudo python setup.py install

If you want a tarball, please look at my email 09/19/2009 09:36 PM -0700.

Ok, I'll install 0.4 using one of these methods. What should I do about the easy_installed stdeb 0.3? Should I try to "remove" 0.3 (I use that term loosely wrt easy_install) using -m?

Ok, I d/l the tarball which installed 0.3-dh7. But this created new problems: Unmet dependencies: debhelper >= 7 python-support >= 0.8.7 Installed: debhelper6.0.4 python-support0.7.5 It's going to be tough to get stdeb running on Ubuntu 8.0.4 LTS. Regards, Gerry So I ran apt-get upgrade

Andrew Straw wrote:

Gerry Reno wrote:

...

It's going to be tough to get stdeb running on Ubuntu 8.0.4 LTS.

Well, I'd qualify that with stdeb version 0.4 may be more tough to get running. It is the bleeding edge, after all. :) 0.3 works as well as it ever did, but new features are going into 0.4.

But now that you mention it, I have backported debhelper and python-support to Hardy. See http://debs.astraw.com/hardy you can download the binaries or source packages from there. (You can also add my repo to your /etc/apt/sources.list)

Wow, there's quite a few packages there. Is debhelper and python-support all that are needed or do I need any others because of dependencies? Regards, Gerry

Andrew Straw wrote:

Gerry Reno wrote:

...

Andrew Straw wrote:

...

Gerry Reno wrote:

...

It's going to be tough to get stdeb running on Ubuntu 8.0.4 LTS.

Well, I'd qualify that with stdeb version 0.4 may be more tough to get running. It is the bleeding edge, after all. :) 0.3 works as well as it ever did, but new features are going into 0.4.

But now that you mention it, I have backported debhelper and python-support to Hardy. See http://debs.astraw.com/hardy you can download the binaries or source packages from there. (You can also add my repo to your /etc/apt/sources.list)

Wow, there's quite a few packages there. Is debhelper and python-support all that are needed or do I need any others because of dependencies?

I can't remember -- you'll have to attempt installation to find out. I think there are some but not too many dependencies that need to be upgraded (I guess on the order of 5 packages or so).

Ok, here is the simplest way that I got stdeb working on Hardy: $ easy_install stdeb # installs 0.3 copied stdeb egg to workarea and edited util.py and removed all "--single-version-externally-managed" options repackaged egg copied egg back to site-packages Now dpkg-buildpackage is happy and no more complaints about unrecognized option or missing dependencies. So my question is this: can you make a special version of the current stdeb 0.3 without the --single-version-externally-managed options and put it into your hardy repo? This would be much easier and less worrisome to hardy users than trying to get them to upgrade a lot of different packages. Regards, Gerry

Andrew Straw wrote:

Gerry Reno wrote:

...

Ok, here is the simplest way that I got stdeb working on Hardy: $ easy_install stdeb # installs 0.3 copied stdeb egg to workarea and edited util.py and removed all "--single-version-externally-managed" options repackaged egg copied egg back to site-packages

OK, but now your packages built with stdeb going to be (even more) different from standard Debian/Ubuntu Python packages -- they're going to install their files as an .egg (directory or zip file). For example, do "dpkg --contents python-my-package_1.0-1.deb" and compare that to standard python packages.

Ok, I just built a .deb using modified stdeb 0.3 on hardy and looked at the contents and it's like this: ./usr/share/pyshared/myapp/dir1/...(list of files) ./usr/share/pyshared/myapp/dir2/... ./usr/share/pyshared/myapp/dir3/... I don't see anything about egg. And it looks very similar to other .deb in /var/cache that I checked. How much difference would I see if it was egg? Regards, Gerry

Andrew, Ok, .deb is building fine using modified stdeb 0.3 on hardy. I've installed the .deb and it works. I can see that stdeb is using debhelper which I really don't know much about. So it looks like debhelper or dh_make is generating the control files. What I need is someway that in myapp maybe under a debian/ directory, I can create postinst and prerm files and then stdeb would look for those in the orig tarball and use them if found. Is this possible? Regards, Gerry

Andrew Straw wrote:

Gerry Reno wrote:

...

Andrew, Ok, .deb is building fine using modified stdeb 0.3 on hardy. I've installed the .deb and it works.

I can see that stdeb is using debhelper which I really don't know much about. So it looks like debhelper or dh_make is generating the control files. What I need is someway that in myapp maybe under a debian/ directory, I can create postinst and prerm files and then stdeb would look for those in the orig tarball and use them if found. Is this possible?

The debian packaging machinery invoked by dpkg-buildpackage will pick up any scripts named my-package-name.preinst, my-package-name.postinst, my-package-name.prerm, my-package-name.postrm that happen to be in the debian/ directory. See http://sdn.vlsm.org/share/Debian-Doc/manuals/maint-guide/ch-dother.en.html for more info. So you have to modify stdeb to drop the appropriate files in there if they're given as a config argument to stdeb. The stdeb master branch already creates such a file for housekeeping reasons (see commit http://github.com/astraw/stdeb/commit/9faaf049a1a7fbd8f5e1fb658248013f208f39... where I added this).

I really didn't have to use any stdeb config args or anything. I just created a dir myapp/debian and put a python-myapp.postinst script file in there, ran my bdist_deb command and then installed the .deb and the python-myapp.postinst script ran perfectly. So do I need to be concerned with config args to stdeb? I do see some python-myapp.postinst.debhelper etc. scripts in there but I don't know if those ran. Regards, Gerry

Gerry Reno wrote:

Gerry Reno wrote:

...

I really didn't have to use any stdeb config args or anything. I just created a dir myapp/debian and put a python-myapp.postinst script file in there, ran my bdist_deb command and then installed the .deb and the python-myapp.postinst script ran perfectly. So do I need to be concerned with config args to stdeb? I do see some python-myapp.postinst.debhelper etc. scripts in there but I don't know if those ran.

So now everything is working on hardy except for the fact that stdeb 0.3 has to be modified to remove the --single-version-externally-managed option. So can you just create a 0.3a version with this small change and that could go out to the repositories and be available by default to everyone. There does not seem to be any impact that I can see in making this change and since so many people create their own install classes this would actually prevent conflicts.

I figured out what the reason for --single-version-externally-managed is -- it is required for namespace packages to function properly. Thus, I can't remove it, as doing so would be a regression. However, if you create a patch I can apply against the old-stable branch that makes this optional, (e.g. set in a similar way as the --ignore-install-requires option), I can add that and make release 0.3.1. Note also that I'm very close to releasing 0.4, as the 0.3 line has major problems with Jaunty and later. Are you planning to submit a patch for the bdist_deb command? I could include that, too. -Andrew

Gerry Reno wrote:

Andrew Straw wrote:

...

Gerry Reno wrote:

...

So now everything is working on hardy except for the fact that stdeb 0.3 has to be modified to remove the --single-version-externally-managed option. So can you just create a 0.3a version with this small change and that could go out to the repositories and be available by default to everyone. There does not seem to be any impact that I can see in making this change and since so many people create their own install classes this would actually prevent conflicts.

I figured out what the reason for --single-version-externally-managed is -- it is required for namespace packages to function properly. Thus, I can't remove it, as doing so would be a regression.

However, if you create a patch I can apply against the old-stable branch that makes this optional, (e.g. set in a similar way as the --ignore-install-requires option), I can add that and make release 0.3.1. Ok, I'll look at getting you a patch that removes the option.

Patch to utils.py in old-stable branch attached.

...

Note also that I'm very close to releasing 0.4, as the 0.3 line has major problems with Jaunty and later.

Are you planning to submit a patch for the bdist_deb command? I could include that, too. Yes, I have something ready but I think the 'bdist_deb' command belongs as a distutils patch. I created the command by subclassing the distutils Command class. Maybe Tarek could comment on what might be best here. Here is my working 'bdist_deb' code verbatim from inside my setup.py: (it could be more pythonic of course with less os.system commands but it's working real well this way at least)

================================================= # Copyright 2009 Gerry Reno. # stdeb note: # stdeb 0.3 needs its util.py modified to remove the install # option "--single-version-externally-managed" otherwise # if you have subclassed the distutils install class to # create your own "install" class (rather common) you will # get an "unrecognized option" error when using 'bdist_deb'. # I had to make this change to stdeb in order to get stdeb to work # on Ubuntu Hardy with a setup.py that defined its own install.

from distutils.core import Command

if sys.hexversion >= 0x020500F0: import stdeb elif sys.argv[1] == 'bdist_deb': print sys.argv[1], "requires python >= 2.5" raise SystemExit(1)

class bdist_deb(Command): # creates .deb pkg from a .dsc file that is generated using stdeb module # requires stdeb, dpkg-dev, build-essential, python-all-dev, debhelper, python-support, fakeroot # # to add 'postinst' and friends just create a debian/ control directory in the home directory of your # application and add a file 'python-myapp.postinst' and it will be run post package installation.

description = 'Create debian-style source (.dsc using stdeb module) and binary (.deb) packages'

user_options = [ ('install-dir=', 'd', "directory to install the files to"), ]

def initialize_options (self): self.install_dir = None self.outfiles = []

def finalize_options (self): pass

def run(self): import setuptools import commands

# run a sub_command (gen .dsc source pkg using stdeb) self.run_command('sdist_dsc')

# save current directory setup_dir = os.getcwd()

# define system command to execute (find source pkg work tree) syscmd = ['find', 'deb_dist', '-mindepth', '1', '-maxdepth', '1', '-type', 'd',]

# execute system command and read output (execute and read output of find cmd) dsc_tree = strip(subprocess.Popen(syscmd, stdout=subprocess.PIPE).communicate()[0]) # strip needed to remove CR

# use system command output (change directory to work tree) os.chdir(dsc_tree)

# define system command to execute (gen .deb binary pkg) syscmd = 'dpkg-buildpackage -rfakeroot -uc -us'

# execute system command if os.system(syscmd) == 0: print '.deb successfully created in deb_dist/' else: print 'problem creating .deb'

# (change directory back to setup dir) os.chdir(setup_dir)

setup(name = name, ...

# add new bdist_deb command here cmdclass = { ... 'bdist_deb' : bdist_deb, ... },

... ) =================================================

Regards, Gerry

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

Andrew Straw wrote:

I saw your patch, but that wasn't what I meant. Disabling --single-version-externally-managed has to be optional, and the argument must continue to be used by default. Simply removing the call will create a regression for namespace packages. Thus, the --single-version-externally-managed should only be removed when someone explicitly asks for it to be removed (because they presumably know what they're doing or are at least aware it may have negative consequences). Hence my suggestion to look at how the --ignore-install-requires works as an example of how to pass an option via the distuils infrastructure.

I'll look at --ignore-install-requires and send another patch.

OK, based on your code I added a "bdist_deb" distutils command to stdeb. I've checked this into the old-stable branch and I'd appreciate it if you can check whether this works for you and send me any comments. Invoke it like this:

python -c "import stdeb; excecfile('setup.py')" bdist_deb

See this was my point of why my 'bdist_deb' belonged as a distutils patch and not in stdeb. The inherited capabilities, behaviors and invocation style needs to follow all the other 'bdist' command types. And that means being able to invoke it the same as say 'bdist_rpm' like: python setup.py bdist_deb or python setup.py bdist_deb --<std bdist type arg> ... If we start allowing 'bdist' derivative commands to diverge in their inherited capabilities, invocation style, and expected behaviors then chaos will ensue. Regards, Gerry

Andrew Straw wrote:

Gerry Reno wrote:

...

Andrew Straw wrote:

...

I saw your patch, but that wasn't what I meant. Disabling --single-version-externally-managed has to be optional, and the argument must continue to be used by default. Simply removing the call will create a regression for namespace packages. Thus, the --single-version-externally-managed should only be removed when someone explicitly asks for it to be removed (because they presumably know what they're doing or are at least aware it may have negative consequences). Hence my suggestion to look at how the --ignore-install-requires works as an example of how to pass an option via the distuils infrastructure.

I'll look at --ignore-install-requires and send another patch.

OK, great.

...

...

OK, based on your code I added a "bdist_deb" distutils command to stdeb. I've checked this into the old-stable branch and I'd appreciate it if you can check whether this works for you and send me any comments. Invoke it like this:

python -c "import stdeb; excecfile('setup.py')" bdist_deb

See this was my point of why my 'bdist_deb' belonged as a distutils patch and not in stdeb. The inherited capabilities, behaviors and invocation style needs to follow all the other 'bdist' command types. And that means being able to invoke it the same as say 'bdist_rpm' like:

python setup.py bdist_deb or python setup.py bdist_deb --<std bdist type arg> ...

If we start allowing 'bdist' derivative commands to diverge in their inherited capabilities, invocation style, and expected behaviors then chaos will ensue.

This does have the same invocation style -- if you change your setup.py to import stdeb. Otherwise, it's still the same invocation style -- you just have to import stdeb outside the setup.py file.

Also, this version of bdist_deb is never going to make it into the standard library as it depends on stdeb.

So feel free to make a counter proposal, but I'm inclined to think this is useful to people as-is and has the benefit of working today.

My view is this. The 'bdist' family of commands should be reserved for pure Distutils implementations not dependent upon setuptools or stdeb. Putting a command to create the .deb based on my code into stdeb is fine but that command should not be a 'bdist_xxx' command. It should be somthing like bstdeb_deb so that a true 'bdist_deb' can still be developed for Distutils and not be in conflict with some other 'bdist_deb' out there. At some point, the 'bdist' family of commands should evolve into OS-specific varieties. And this means that for example, redhat, fedora, mandriva, or any other rpm-package distro would have their own 'bdist_DISTRO' command that would inherit from 'bdist_rpm' and then institute their own rules/policies onto a generic 'bdist_rpm'. And the same would hold true for example a 'bdist_ubuntu' command that would inherit from a 'bdist_deb' command or in general for any 'bdist_DISTRO' command that would inherit from a 'bdist_PACKAGETOOL' command. Maybe we can get some overall guidance here from Tarek who has been coordinating a lot of these types of things. Regards, Gerry

Andrew Straw wrote:

...

...

...

...

OK, based on your code I added a "bdist_deb" distutils command to stdeb. I've checked this into the old-stable branch and I'd appreciate it if you can check whether this works for you and send me any comments. Invoke it like this:

python -c "import stdeb; excecfile('setup.py')" bdist_deb

See this was my point of why my 'bdist_deb' belonged as a distutils patch and not in stdeb. The inherited capabilities, behaviors and invocation style needs to follow all the other 'bdist' command types. And that means being able to invoke it the same as say 'bdist_rpm' like:

python setup.py bdist_deb or python setup.py bdist_deb --<std bdist type arg> ...

If we start allowing 'bdist' derivative commands to diverge in their inherited capabilities, invocation style, and expected behaviors then chaos will ensue.

This does have the same invocation style -- if you change your setup.py to import stdeb. Otherwise, it's still the same invocation style -- you just have to import stdeb outside the setup.py file.

Also, this version of bdist_deb is never going to make it into the standard library as it depends on stdeb.

So feel free to make a counter proposal, but I'm inclined to think this is useful to people as-is and has the benefit of working today.

My view is this. The 'bdist' family of commands should be reserved for pure Distutils implementations not dependent upon setuptools or stdeb. Putting a command to create the .deb based on my code into stdeb is fine but that command should not be a 'bdist_xxx' command. It should be somthing like bstdeb_deb so that a true 'bdist_deb' can still be developed for Distutils and not be in conflict with some other 'bdist_deb' out there.

Well, it wouldn't be setting a precedent that importing a module would extend the functionality of the stdlib's distutils commands -- the setuptools and distribute projects already do this. For example, the "sdist" command in setuptools has many differences with that of "sdist" in plain distutils. The point is that a user can choose not to import setuptools and thus not get the different behavior.

The case here is that if the stdlib ever grows bdist_deb, that's fine, but I don't see why that should prevent stdeb from implementing a bdist_deb command of its own. In this case, I think the issue is even less relevant, since there is no stdlib "bdist_deb" command. Users would naturally expect "bdist_deb" to do exactly what stdeb is implementing. As long as I'm going ahead an implementing the functionality (which I'm ambivalent about in and of itself for previously stated reasons), I think it should be named the most obvious name.

My only concern is that namespaces are important and if we end up with two solution paths that each use the same namespace then this becomes very confusing for users. Yes, I understand that if you don't import stdeb in your setup.py then you can use a 'bdist_deb' from Distutils.

...

At some point, the 'bdist' family of commands should evolve into OS-specific varieties. And this means that for example, redhat, fedora, mandriva, or any other rpm-package distro would have their own 'bdist_DISTRO' command that would inherit from 'bdist_rpm' and then institute their own rules/policies onto a generic 'bdist_rpm'. And the same would hold true for example a 'bdist_ubuntu' command that would inherit from a 'bdist_deb' command or in general for any 'bdist_DISTRO' command that would inherit from a 'bdist_PACKAGETOOL' command.

I don't see what's so specific about ubuntu versus debian right now that there's any reason for a different set of commands for the two. Therefore, I think this is a red herring w.r.t. the present discussion.

For ubuntu, at the moment at least, that's true. But distros like redhat are creating their own rules/policies around rpm naming conventions, build options, etc. and of course they should be able to do this and I expect other distros will do the same, and these will diverge more over time even though they will all use the same packaging tool underneath.

...

Maybe we can get some overall guidance here from Tarek who has been coordinating a lot of these types of things.

I think this is a good topic for someone with a long view about where things are headed here to weigh in on this. I'm fine with whatever fits in with the longer term goals. Regards, Gerry

Andrew, Here's another patch attempt. Is this what you had in mind?

Regards, Gerry OK, I looked at this, and now I'm throughly confused by what you are

Gerry Reno wrote: trying to do. Back to your big picture, as I understood it, you were trying to avoid importing setuptools because you derive some distutils commands and you don't want to derive from the setuptools versions. And the --single-version-externally-managed was giving you problems because that's not supported by distutils. But A) I can't see how this patch would address the issue of setuptools not being imported, because that still happens. And B) the patch as you sent it was incomplete, requiring more work on my part even to be able to get it to run without raising exceptions. So I'm not sure that I actually "get" what your patch does here. I attempted to do my best and have now started a branch for this patch and subsequent efforts. See http://github.com/astraw/stdeb/tree/gerry-reno . My testing on this issue has been based on the version in this branch. Please either submit any patches off the old-stable branch or extending this gerry-reno branch. -Andrew

2009/9/23 Andrew Straw :

Well, it wouldn't be setting a precedent that importing a module would extend the functionality of the stdlib's distutils commands -- the setuptools and distribute projects already do this. For example, the "sdist" command in setuptools has many differences with that of "sdist" in plain distutils. The point is that a user can choose not to import setuptools and thus not get the different behavior.

The case here is that if the stdlib ever grows bdist_deb, that's fine, but I don't see why that should prevent stdeb from implementing a bdist_deb command of its own. In this case, I think the issue is even less relevant, since there is no stdlib "bdist_deb" command. Users would naturally expect "bdist_deb" to do exactly what stdeb is implementing. As long as I'm going ahead an implementing the functionality (which I'm ambivalent about in and of itself for previously stated reasons), I think it should be named the most obvious name.

During last Pycon summit, we v'e said that part of the problem with Distutils was that it was doing too much things and that we should probably try to remove all platform-specific commands to reduce its scope. For instance, bdist_rpm is fine, but being located in Python stdlib, its release cycle is too long, and it would probably be better for it to be on its own, managed by people from the "rpm community" (if this term makes sense, e.g. Fedora guys maybe ?) Same thing goes with bdist_deb : while it's more controversial to remove existing bdist_* command, we will not include new bdist_* commands for sure in Distutils, and rather let the debian/ubuntu community provide the best packaging solution on the top of Distutils. That said, if stdeb provides a plain Distutils command, that doesn't use Setuptools and if that command is stable and used in the community, we will be happy to add a pointer to it in Distutils documentation. The problem of using Setuptools or Distribute in a command is that it does a lot of patching (much more that overriding distutils sdist command) and that changes Distutils global behavior. Notice that the Distribute project plans to remove that patching in the future. Notice also that Distutils has a plugin system, where you can point a package that contains extra commands. This configuration can be done in distutils.cfg. This allows the usage of the command in the CLI without having to import the command directly in the setup.py file. Last, about the name, I think using "bdist_deb" is fine imho. If another project uses the same name, it's juste a matter of configuring the one to be used in a given context. Regards Tarek -- Tarek Ziadé | http://ziade.org | オープンソースはすごい!

Gerry Reno wrote:

Tarek Ziadé wrote:

...

2009/9/23 Andrew Straw :

...

Well, it wouldn't be setting a precedent that importing a module would extend the functionality of the stdlib's distutils commands -- the setuptools and distribute projects already do this. For example, the "sdist" command in setuptools has many differences with that of "sdist" in plain distutils. The point is that a user can choose not to import setuptools and thus not get the different behavior.

The case here is that if the stdlib ever grows bdist_deb, that's fine, but I don't see why that should prevent stdeb from implementing a bdist_deb command of its own. In this case, I think the issue is even less relevant, since there is no stdlib "bdist_deb" command. Users would naturally expect "bdist_deb" to do exactly what stdeb is implementing. As long as I'm going ahead an implementing the functionality (which I'm ambivalent about in and of itself for previously stated reasons), I think it should be named the most obvious name.

During last Pycon summit, we v'e said that part of the problem with Distutils was that it was doing too much things and that we should probably try to remove all platform-specific commands to reduce its scope.

For instance, bdist_rpm is fine, but being located in Python stdlib, its release cycle is too long, and it would probably be better for it to be on its own, managed by people from the "rpm community" (if this term makes sense, e.g. Fedora guys maybe ?)

Same thing goes with bdist_deb : while it's more controversial to remove existing bdist_* command, we will not include new bdist_* commands for sure in Distutils, and rather let the debian/ubuntu community provide the best packaging solution on the top of Distutils.

That said, if stdeb provides a plain Distutils command, that doesn't use Setuptools and if that command is stable and used in the community, we will be happy to add a pointer to it in Distutils documentation.

The problem of using Setuptools or Distribute in a command is that it does a lot of patching (much more that overriding distutils sdist command) and that changes Distutils global behavior. Notice that the Distribute project plans to remove that patching in the future.

Notice also that Distutils has a plugin system, where you can point a package that contains extra commands. This configuration can be done in distutils.cfg.

This allows the usage of the command in the CLI without having to import the command directly in the setup.py file.

Last, about the name, I think using "bdist_deb" is fine imho. If another project uses the same name, it's juste a matter of configuring the one to be used in a given context.

Regards Tarek

So are you saying then that a stddeb-based 'bdist_deb' should be implemented as a standalone plugin to Distribute? And how would that work for distros such as Ubuntu 8.0.4 Hardy? Would they be able to take advantage of Distribute?

I think a long term goal should be getting away from the plugin nature of distutils. I don't see why distutils (or my mythical distlib) shouldn't just be a package (or more likely multiple packages) that stand-alone tools can use. Other than much complexity and confusion, I don't see what plugins have bought us. Not everything (and especially not distutils) needs to be a framework. I'll admit that this plan in years down the road, at best. And it's not anyone's official position. Eric.

Tarek Ziadé wrote:

Gerry:

...

...

So are you saying then that a stddeb-based 'bdist_deb' should be implemented as a standalone plugin to Distribute? And how would that work for distros such as Ubuntu 8.0.4 Hardy? Would they be able to take advantage of Distribute?

Yes, it's just a matter of packaging this command in a standalone distribution. Once it's installed, and distutils.cfg properly configured, it can be used as a plugin.

Distribute is not yet a good citizen for being used in such packages, because it still patches distutils like setuptools does.

Eric:

...

I think a long term goal should be getting away from the plugin nature of distutils. I don't see why distutils (or my mythical distlib) shouldn't just be a package (or more likely multiple packages) that stand-alone tools can use. Other than much complexity and confusion, I don't see what plugins have bought us. Not everything (and especially not distutils) needs to be a framework.

It's just a matter of defining *where* is the CLI and how it works. The plugin system here is just to make those commands reachable from distutils's setup.py machinery. If another package has its own CLI, it can ignore this plugin system I guess.

...

I'll admit that this plan in years down the road, at best. And it's not anyone's official position.

Agreed, we need to think about alternatives.

but in any case, nowadays, if the stdeb project wrap its work in a bdist_deb command that can be used as a distutils plugin, I don't think it's a problem for the future.

I would be pretty happy about it :)

Tarek

Then I'm ok then with having 'bdist_deb' be the name and using the plugin approach. Andrew, what do you think about this solution? Regards, Gerry

Andrew Straw wrote:

Gerry Reno wrote:

...

Then I'm ok then with having 'bdist_deb' be the name and using the plugin approach.

Andrew, what do you think about this solution?

It's already committed as of a couple days ago in the "old-stable" branch and I just merged it into the master branch.

I'll release 0.3.1 (from the old-stable branch) and 0.4 (from the master branch) with this soon.

-Andrew

In reviewing the gerry-reno git branch and the old-stable branch I don't see where there are any user-options declared in stdeb/command/bdist_deb.py. Shouldn't the same user-options be there as are in sdist_dsc.py? The 'bdist_deb' command usefulness would be very limited without the ability to pass options to it and hence to 'sdist_dsc'. Regards, Gerry

Andrew Straw wrote:

Gerry Reno wrote:

...

Andrew Straw wrote:

...

Gerry Reno wrote:

...

Then I'm ok then with having 'bdist_deb' be the name and using the plugin approach.

Andrew, what do you think about this solution?

It's already committed as of a couple days ago in the "old-stable" branch and I just merged it into the master branch.

I'll release 0.3.1 (from the old-stable branch) and 0.4 (from the master branch) with this soon.

-Andrew

In reviewing the gerry-reno git branch and the old-stable branch I don't see where there are any user-options declared in stdeb/command/bdist_deb.py. Shouldn't the same user-options be there as are in sdist_dsc.py? The 'bdist_deb' command usefulness would be very limited without the ability to pass options to it and hence to 'sdist_dsc'.

What options do you want to pass to it? bdist_deb does very little other than call dpkg-buildpackage on the result of sdist_dsc.

As far as how -- that's standard distutils. You should be able to follow the lead from the sdist_dsc command, for example.

Well, for starters, we need to be able to pass the option '--ignore-single-version-externally-managed' and beyond that we need to pass other options down to sdist_dsc, things like '--ignore-install-requires'. Regards, Gerry

Andrew Straw wrote:

Gerry Reno wrote:

...

Andrew Straw wrote:

...

Gerry Reno wrote:

...

Andrew Straw wrote:

...

Gerry Reno wrote:

...

Then I'm ok then with having 'bdist_deb' be the name and using the plugin approach.

Andrew, what do you think about this solution?

It's already committed as of a couple days ago in the "old-stable" branch and I just merged it into the master branch.

I'll release 0.3.1 (from the old-stable branch) and 0.4 (from the master branch) with this soon.

-Andrew

In reviewing the gerry-reno git branch and the old-stable branch I don't see where there are any user-options declared in stdeb/command/bdist_deb.py. Shouldn't the same user-options be there as are in sdist_dsc.py? The 'bdist_deb' command usefulness would be very limited without the ability to pass options to it and hence to 'sdist_dsc'.

What options do you want to pass to it? bdist_deb does very little other than call dpkg-buildpackage on the result of sdist_dsc.

As far as how -- that's standard distutils. You should be able to follow the lead from the sdist_dsc command, for example.

Well, for starters, we need to be able to pass the option '--ignore-single-version-externally-managed' and beyond that we need to pass other options down to sdist_dsc, things like '--ignore-install-requires'.

Just pass the arguments directly to sdist_dsc. It should be something like this:

python setup.py sdist_dsc --ignore-single-version-externally-managed --ignore-install-requires bdist_deb

How's that going to work? You have 'sdist_dsc' on the command line as well as 'bdist_deb'. 'bdist_deb' also calls 'sdist_dsc' internally but this time it would be without any arguments? How can that work? Regards, Gerry

2009/9/28 Gerry Reno :

...

...

Just pass the arguments directly to sdist_dsc. It should be something like this:

python setup.py sdist_dsc --ignore-single-version-externally-managed --ignore-install-requires bdist_deb

How's that going to work? You have 'sdist_dsc' on the command line as well as 'bdist_deb'. 'bdist_deb' also calls 'sdist_dsc' internally but this time it would be without any arguments? How can that work?

This is standard distutils command processing -- this has nothing to do with stdeb per se. The reason I gave an example not involving stdeb (which was snipped out by you in your quote above) is so that you can

Olof Bjarnason wrote: try it yourself and explore further without thinking it's somehow an stdeb issue. Specifically, you do not need to understand anything specific to stdeb to understand how to pass multiple arguments to sub-commands of a distutils command. (And if stdeb handles things differently, it's a bug and I would appreciate hearing about it.) For example, "build_ext" is a sub-command of "install" that get called to build extension modules prior to installation. (Being a sub-command is not strictly true in the sense that the install command doesn't list build_ext in its sub_commands in distutils/commands/install.py, but it acts as if it's true.) Likewise, "sdist_dsc" is a sub-command of "bdist_deb". In all cases one, might want to pass arguments to the sub-command before running a later command. distutils allows this.

...

OK I know I'm a noob on *nix distribution of python apps', but:

Would it be possible to extend setup.py with info for package dependency?

Then python setup.py sdist_deb would know what package dependencies it needs to add to the resulting .deb file.

Read the README.rst that is shipped with stdeb, specifically the section "Customizing the produced Debian source package (config options)". You want the "Depends" field in the stdeb.cfg file. (This will presumably be back online at the stdeb homepage at http://github.com/astraw/stdeb also, once github's move to rackspace is complete.)

There is also some stuff to automatically attempt to find dependencies based on the setuptools install_requires fields. -Andrew

2009/9/28 Gerry Reno :

Andrew Straw wrote:

...

Olof Bjarnason wrote:

...

2009/9/28 Gerry Reno :

...

...

Just pass the arguments directly to sdist_dsc. It should be something like this:

python setup.py sdist_dsc --ignore-single-version-externally-managed --ignore-install-requires bdist_deb

How's that going to work?  You have 'sdist_dsc' on the command line as well as 'bdist_deb'.   'bdist_deb' also calls 'sdist_dsc' internally but this time it would be without any arguments?  How can that work?

This is standard distutils command processing -- this has nothing to do with stdeb per se. The reason I gave an example not involving stdeb (which was snipped out by you in your quote above) is so that you can try it yourself and explore further without thinking it's somehow an stdeb issue. Specifically, you do not need to understand anything specific to stdeb to understand how to pass multiple arguments to sub-commands of a distutils command. (And if stdeb handles things differently, it's a bug and I would appreciate hearing about it.)

For example, "build_ext" is a sub-command of "install" that get called to build extension modules prior to installation. (Being a sub-command is not strictly true in the sense that the install command doesn't list build_ext in its sub_commands in distutils/commands/install.py, but it acts as if it's true.) Likewise, "sdist_dsc" is a sub-command of "bdist_deb". In all cases one, might want to pass arguments to the sub-command before running a later command. distutils allows this.

Ok, the commands behave like makefile rules, once run they don't run again. But there are still several issues here:

Remember that I said that my goal with 'bdist_deb' was for users to have a SINGLE command to generate a .deb.

What needs to be achieved is for a command like this: $ python setup.py  bdist_deb OR $ python setup.py  bdist_deb  --ignore-single-version-externally-managed

to be possible.  'bdist_deb' would call sdist_dsc internally with the necessary args WITHOUT having to explicitly put 'sdist_dsc' on the command line.

This would be _exactly_ what I'm looking for :) But how would you know what package dependencies the python application has? Also, would it be a binary or source distribution package..?

This is how I was expecting that you would implement the bdist_deb command inside of stdeb.

Another issue:  In util.py on line 962 (gerry-reno git): %(setup_env_vars)spython$* -c "import setuptools,sys;f='setup.py';sys.argv[0]=f;execfile(f,{'__file__':f,'__name__':'__main__'})" install \\

Maybe it should be like this (note semicolon splitting line into two statements): %(setup_env_vars)s;    python$* -c "import setuptools,sys;f='setup.py';sys.argv[0]=f;execfile(f,{'__file__':f,'__name__':'__main__'})" install \\

Otherwise any of the env vars don't seem to be in effect for the  line itself.   If you put any of the env vars in the line, they don't expand to the value set in set_env_vars.  They expand to previous value or to null.

Regards, Gerry

_______________________________________________ Distutils-SIG maillist  -  Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

-- twitter.com/olofb olofb.wordpress.com olofb.wordpress.com/tag/english

Gerry Reno wrote:

Olof Bjarnason wrote:

...

...

Ok, the commands behave like makefile rules, once run they don't run again. But there are still several issues here:

Remember that I said that my goal with 'bdist_deb' was for users to have a SINGLE command to generate a .deb.

What needs to be achieved is for a command like this: $ python setup.py bdist_deb OR $ python setup.py bdist_deb --ignore-single-version-externally-managed

to be possible. 'bdist_deb' would call sdist_dsc internally with the necessary args WITHOUT having to explicitly put 'sdist_dsc' on the command line.

This would be _exactly_ what I'm looking for :)

I know. I've had many of my users requesting exactly the same thing as well and I've been pursuing this for months now and when I got a 'bdist_deb' working with stdeb I knew I was getting close. It just needs a little more tweeking and I think we'll have it. Let's wait and see what Andrew says here. I don't understand what you're waiting on me for at this point.

Olaf -- as I understand it -- you need to work with Ubuntu 9.04. I just released stdeb 0.4 which supports this "bdist_deb" and doesn't pass the --single-version-externally-managed option anyway and therefore doesn't support the "--ignore-single-version-externally-managed" option. I don't understand why you need to pass "--ignore-single-version-externally-managed". "python setup.py bdist_deb" should work for you. Please report with specific information about expected and actual behavior if things are not working to your desire (taking care to read the, admittedly minimal, documentation in README.rst). Gerry -- I don't understand why you want to subvert the normal distutils way of doing things. Passing arguments to bdist_deb that are really arguments to sdist_dsc just isn't the way distutils does things. You're asking for the equivalent of being able to pass arguments to the distutils install command that are ultimately intended for the distutils build_ext command. If you want to test (and possibly implement) functionality such as adding a [sdist_dsc] section to setup.cfg where these options could just live so that you don't need to pass arguments at the command line, thats would be welcome. Gerry, point 2 -- I still think you're not looking at your own big picture here -- AFAIK, you are bending over backwards to attempt to pass the "--ignore-single-version-externally-managed" so that you don't have to import setuptools in your setup.py file to avoid the monkeypatching that setuptools does. But, setuptools is explicitly imported in the debian/rules files created in stdeb 0.3.x. I get the impression that you're fixated on this argument passing business and not looking at the bigger picture of the architecture involved and whether your ultimate goal is even achievable with stdeb 0.3.x. -Andrew

Andrew Straw wrote:

...

Gerry Reno wrote:

...

Olof Bjarnason wrote:

...

...

Ok, the commands behave like makefile rules, once run they don't run again. But there are still several issues here:

Remember that I said that my goal with 'bdist_deb' was for users to have a SINGLE command to generate a .deb.

What needs to be achieved is for a command like this: $ python setup.py bdist_deb OR $ python setup.py bdist_deb --ignore-single-version-externally-managed

to be possible. 'bdist_deb' would call sdist_dsc internally with the necessary args WITHOUT having to explicitly put 'sdist_dsc' on the command line.

This would be _exactly_ what I'm looking for :)

I know. I've had many of my users requesting exactly the same thing as well and I've been pursuing this for months now and when I got a 'bdist_deb' working with stdeb I knew I was getting close. It just needs a little more tweeking and I think we'll have it. Let's wait and see what Andrew says here.

I don't understand what you're waiting on me for at this point.

Olaf -- as I understand it -- you need to work with Ubuntu 9.04. I just released stdeb 0.4 which supports this "bdist_deb" and doesn't pass the --single-version-externally-managed option anyway and therefore doesn't support the "--ignore-single-version-externally-managed" option. I don't understand why you need to pass "--ignore-single-version-externally-managed". "python setup.py bdist_deb" should work for you. Please report with specific information about expected and actual behavior if things are not working to your desire (taking care to read the, admittedly minimal, documentation in README.rst).

Gerry -- I don't understand why you want to subvert the normal distutils way of doing things. Passing arguments to bdist_deb that are really arguments to sdist_dsc just isn't the way distutils does things. Please explain this then. Why EVEN bother to call 'sdist_dsc' from within 'bdist_deb' if you cannot somehow pass arguments to the internal 'sdist_dsc' call? You can. "python setup.py commandA --option-for-command-A commandB --option-for-command-B". This is just how distutils works. This is not an stdeb issue. If you don't need to pass options to commandA, but commandA is still a sub-command of commandB, then commandA will still get called, just without any options. As I said, you can also pass

Gerry Reno wrote: options by placing them in the appropriate setup.cfg location. For example: python setup.py build_py --build-lib=zzz bdist_dumb is exactly equivalent to python setup.py bdist_dumb with if setup.cfg file contains: [build_py] build-lib=zzz Because build_py A) is a sub-command of bdist_dumb and therefore gets run and B) picks up its options using the standard distutils mechanisms, either from the command line or setup.cfg.

...

You're asking for the equivalent of being able to pass arguments to the distutils install command that are ultimately intended for the distutils build_ext command. If you want to test (and possibly implement) functionality such as adding a [sdist_dsc] section to setup.cfg where these options could just live so that you don't need to pass arguments at the command line, thats would be welcome.

Gerry, point 2 -- I still think you're not looking at your own big picture here -- AFAIK, you are bending over backwards to attempt to pass the "--ignore-single-version-externally-managed" so that you don't have to import setuptools in your setup.py file to avoid the monkeypatching that setuptools does. Stop. NO. IF I import stdeb in the setup.py THEN any local distutils install class gets an error: --single-version-externally-managed not recognized (or something similar).

So why don't you just derive from setuptools' install command (rather than distutils' install command) if you insist on using stdeb 0.3? I think this will solve all your problems in a much simpler way.

The only way to get by this is by passing in the NEW option that I made, '--ignore-single-version-externally-managed' which removes this option from the install command which then allows the install to succeed. This is because it looks like stdeb hijacks the distutils install command.

I have yet to see a working patch that I can actually test that backs up your assertion. As it is, I simply don't believe you because the debian/rules created by stdeb 0.3 not only passes that option, but it also explicitly imports setuptools. The setuptools install command does know the --single-version-externally-managed option. (I think in your memory of what worked, you are forgetting that you also removed the "import setuptools".) -Andrew

Gerry Reno wrote:

What if stdeb only had the command 'bdist_deb' and had no other command.

I will not remove the "sdist_dsc" command from stdeb. I believe that the ability to produce debian source packages is much more important that the ability to produce binary packages which only target a single architecture and version of Debian/Ubuntu. "bdist_deb" is just a trivial add-on to that saving a couple lines of typing which may be helpful to people who aren't familiar with those lines. But I see the .dsc and associated files as much more useful as the compiled .deb files.

Everything else was taken care of internally. What then? Are you saying then that it is impossible to pass in any options to 'bdist_deb' to affect the generation of the source package? If true, this makes NO SENSE. I still don't understand why you insist the options are to the bdist_deb command. What is wrong with passing the options to the sdist_dsc command where they take effect anyway?

Yes, I can see that once we got invaded with setuptools it seems you really have no other choice. Hang on a minute. The "st" in stdeb is for "setuptools". While in the 0.4 version this may take a less prominent role, and Distribute might be able to step in, the ontogeny of stdeb is very much tied up with setuptools. So I'm not comfortable promoting the idea that there ever existed stdeb without setuptools. Maybe it can happen in the future -- but it's certainly never happened in the past.

And getting back to the issue that got skipped: Another issue: In util.py on line 962 (gerry-reno git): %(setup_env_vars)spython$* -c "import setuptools,sys;f='setup.py';sys.argv[0]=f;execfile(f,{'__file__':f,'__name__':'__main__'})" install \\

Maybe it should be like this (note semicolon splitting line into two statements): %(setup_env_vars)s; python$* -c "import setuptools,sys;f='setup.py';sys.argv[0]=f;execfile(f,{'__file__':f,'__name__':'__main__'})" install \\

Otherwise any of the env vars don't seem to be in effect for the line itself. If you put any of the env vars in the line, they don't expand to the value set in set_env_vars. They expand to previous value or to null. For setting environment variables this way, they cannot be separated by a semicolon.

Gerry Reno wrote:

Andrew Straw wrote:

...

Gerry Reno wrote:

...

What if stdeb only had the command 'bdist_deb' and had no other command.

I will not remove the "sdist_dsc" command from stdeb. I believe that the ability to produce debian source packages is much more important that the ability to produce binary packages which only target a single architecture and version of Debian/Ubuntu. "bdist_deb" is just a trivial add-on to that saving a couple lines of typing which may be helpful to people who aren't familiar with those lines. But I see the .dsc and associated files as much more useful as the compiled .deb files.

LMAO. Dang it. I am not suggesting or implying that you should remove the 'sdist_dsc' command. I'm trying to illustrate that the commandA-opts are just that and nothing more: commandA-opts. And whatever part of the build/install process that they apply to, then the commandA can use them for WHATEVER purpose. Which means that since 'bdist_deb' calls 'sdist_dsc' it is perfectly ok to allow all 'sdist_dsc' options to be passed into the 'bdist_deb' command which can then pass them along to 'sdist_dsc'. 'bdist_deb' is a SUPERSET of 'sdist_dsc'. I see -- you want to pass-through the options to sdist_dsc. Fine, implement this in a patch and I'll accept it.

...

For setting environment variables this way, they cannot be separated by a semicolon The environment variables are not individually separated by a semicolon. The WHOLE shell script environment variable declaration statement needs to be separated from the install line statement.

Simple demonstration of the problem with the 'echo' statement simulating the 'install' statement: $ a='123' b='456' echo test $a $b test $ a='123' b='456'; echo test $a $b test 123 456

Passing to a subprocess is different. We're passing to a subprocess: $ a='123' python -c "import os; print os.environ['a']" 123 $ a='123'; python -c "import os; print os.environ['a']" Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python2.6/UserDict.py", line 22, in __getitem__ raise KeyError(key) KeyError: 'a'

Gerry Reno wrote:

Andrew Straw wrote:

...

Gerry Reno wrote:

...

Andrew Straw wrote:

...

Gerry Reno wrote:

...

What if stdeb only had the command 'bdist_deb' and had no other command. I will not remove the "sdist_dsc" command from stdeb. I believe that the ability to produce debian source packages is much more important that the ability to produce binary packages which only target a single architecture and version of Debian/Ubuntu. "bdist_deb" is just a trivial add-on to that saving a couple lines of typing which may be helpful to people who aren't familiar with those lines. But I see the .dsc and associated files as much more useful as the compiled .deb files.

LMAO. Dang it. I am not suggesting or implying that you should remove the 'sdist_dsc' command. I'm trying to illustrate that the commandA-opts are just that and nothing more: commandA-opts. And whatever part of the build/install process that they apply to, then the commandA can use them for WHATEVER purpose. Which means that since 'bdist_deb' calls 'sdist_dsc' it is perfectly ok to allow all 'sdist_dsc' options to be passed into the 'bdist_deb' command which can then pass them along to 'sdist_dsc'. 'bdist_deb' is a SUPERSET of 'sdist_dsc'. I see -- you want to pass-through the options to sdist_dsc. Fine, implement this in a patch and I'll accept it. Oh sure, I get to have all the fun.

...

...

...

For setting environment variables this way, they cannot be separated by a semicolon The environment variables are not individually separated by a semicolon. The WHOLE shell script environment variable declaration statement needs to be separated from the install line statement.

Simple demonstration of the problem with the 'echo' statement simulating the 'install' statement: $ a='123' b='456' echo test $a $b test $ a='123' b='456'; echo test $a $b test 123 456

Passing to a subprocess is different. We're passing to a subprocess:

$ a='123' python -c "import os; print os.environ['a']" 123 $ a='123'; python -c "import os; print os.environ['a']" Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python2.6/UserDict.py", line 22, in __getitem__ raise KeyError(key) KeyError: 'a'

Ok, I see what is happening. In order for the env vars to be available to both the line itself AND to the subprocess it has to be declared as exported like this:

export a='123' b='456'; python -c "import os; print os.environ['a']" && echo $b 123 456

Now it will work both ways.

Regards, Gerry

I have attached a replacement 'bdist_deb.py' file that permits passing arguments to bdist_deb which in turn passes them down to sdist_dsc. and a util.py diff (made against the gerry-reno git branch) makes the setup_env_vars work for both the line statements as well as the subprocesses. I have tested both on Ubuntu 8.0.4 Hardy with success. Regards, Gerry 962,963c962,963 < %(setup_env_vars)spython$* -c "import setuptools,sys;f='setup.py';sys.argv[0]=f;execfile(f,{'__file__':f,'__name__':'__main__'})" install \\ < --no-compile ${SVEMOPT} \\ ---

export %(setup_env_vars)s; python$* -c "import setuptools,sys;f='setup.py';sys.argv[0]=f;execfile(f,{'__file__':f,'__name__':'__main__'})" install \\ --no-compile $${SVEMOPT} \\

968,969c968,969 < %(setup_env_vars)spython$* -c "import setuptools,sys;f='setup.py';sys.argv[0]=f;execfile(f,{'__file__':f,'__name__':'__main__'})" install \\ < --no-compile ${SVEMOPT} --install-layout=deb \\ ---

export %(setup_env_vars)s; python$* -c "import setuptools,sys;f='setup.py';sys.argv[0]=f;execfile(f,{'__file__':f,'__name__':'__main__'})" install \\ --no-compile $${SVEMOPT} --install-layout=deb \\

Gerry Reno wrote:

...

I have attached a replacement 'bdist_deb.py' file that permits passing arguments to bdist_deb which in turn passes them down to sdist_dsc.

and a util.py diff (made against the gerry-reno git branch) makes the setup_env_vars work for both the line statements as well as the subprocesses. OK, I committed these to the old-stable branch and merged in the bdist_deb changes to the master branch. I made some minor changes to your option to disable --single-version..., the most significant of which is the name, which is now "--disable..." instead of "--ignore...". The other is that I'm not using env vars in the debian/rules file -- it just clutters it up unnecessarily. Even so, it would still be more flexible if you leave in the 'export' statement. That allows you to use the env vars in both the line AND the subprocess. I know you're not using them right now, but at least later

Andrew Straw wrote: people would have the option if they wanted. And since you refactored the SVEM option you should make sure that there is at least ONE env var for the export statement and that can be anything, such as export HOME PATH DUMMY, etc. It's not absolutely critical but 'export' with no options will print all the exported env vars which might look strange in the log/output.

I still wonder if the (now named) --disable-single-version... option actually solves your problem -- you _are_ importing setuptools in the same line, so I am mystified by this.

Ok, here's what happens if I DON'T use the option (shown with the old code - same logic): $ python setup.py bdist_deb ... running build_scripts ... if test "2.5" = "2.5" -o "2.5" = "2.4"; then \ export SVEMOPT="--single-version-externally-managed" ; python2.5 -c "import setuptools,sys;f='setup.py';sys.argv[0]=f;execfile(f,{'__file__':f,'__name__':'__main__'})" install \ --no-compile ${SVEMOPT} \ ... error: option --single-version-externally-managed not recognized ... <process stops on the error> I can't explain it, but this is what happens. This is on Ubuntu Hardy using python 2.5.2.

But anyhow it's now in there and I don't think it'll cause anyone any harm because it's unlikely they'll use the option.

I'll probably let this stuff settle for a day or two and then cut new stdeb releases (0.3.2 and 0.4.1).

-Andrew

Regards, Gerry

Andrew Straw wrote:

Gerry Reno wrote:

...

Ok, here's what happens if I DON'T use the option (shown with the old code - same logic):

$ python setup.py bdist_deb ... running build_scripts ... if test "2.5" = "2.5" -o "2.5" = "2.4"; then \ export SVEMOPT="--single-version-externally-managed" ; python2.5 -c "import setuptools,sys;f='setup.py';sys.argv[0]=f;execfile(f,{'__file__':f,'__name__':'__main__'})" install \ --no-compile ${SVEMOPT} \ ... error: option --single-version-externally-managed not recognized ... <process stops on the error>

I can't explain it, but this is what happens. This is on Ubuntu Hardy using python 2.5.2.

I see. I think it's because you're subclassing distutils.command.install rather than setuptools.command.install for your install command.

Another idea is that you could also detect whether setuptools is imported by doing something like "if 'setuptools' in sys.modules" and then subclassing the appropriate install command based on that. Then you wouldn't need to pass the --disable-single-version-externally-managed argument at all.

-Andrew

Ok, a little guidance please. As I understand it if I import setuptools into my app setup.py and I subclass my install from setuptools.command.install, then I can also use the option '--single-version-externally-managed' to make the install behave like the distutils style install. Is that correct? If this is true then what I would like to do is to force this option on in the code. How can I do that? I know I can probably put it into setup.cfg but I just think it would be better hardwired into the code. Regards, Gerry

Gerry Reno wrote:

...

...

Gerry Reno wrote:

...

Ok, here's what happens if I DON'T use the option (shown with the old code - same logic):

$ python setup.py bdist_deb ... running build_scripts ... if test "2.5" = "2.5" -o "2.5" = "2.4"; then \ export SVEMOPT="--single-version-externally-managed" ; python2.5 -c "import setuptools,sys;f='setup.py';sys.argv[0]=f;execfile(f,{'__file__':f,'__name__':'__main__'})"

install \ --no-compile ${SVEMOPT} \ ... error: option --single-version-externally-managed not recognized ... <process stops on the error>

I can't explain it, but this is what happens. This is on Ubuntu Hardy using python 2.5.2. I see. I think it's because you're subclassing distutils.command.install rather than setuptools.command.install for your install command.

Another idea is that you could also detect whether setuptools is imported by doing something like "if 'setuptools' in sys.modules" and then subclassing the appropriate install command based on that. Then you wouldn't need to pass the --disable-single-version-externally-managed argument at all.

-Andrew Ok, a little guidance please. As I understand it if I import setuptools into my app setup.py and I subclass my install from setuptools.command.install, then I can also use the option '--single-version-externally-managed' to make the install behave like

Andrew Straw wrote: the distutils style install. Is that correct? If this is true then what I would like to do is to force this option on in the code. How can I do that? I know I can probably put it into setup.cfg but I just think it would be better hardwired into the code.

No one?

Ok, I refactored my app install code and have it running with setuptools.

Andrew, how can I generate the .deb package without the leading 'python-' in the name? Is this configurable? Yes. The 'Package' option. Described in "Customizing the produced Debian

Gerry Reno wrote: source package (config options)" in the README.rst. -Andrew

Gerry Reno wrote:

Andrew Straw wrote:

...

...

Andrew, how can I generate the .deb package without the leading 'python-' in the name? Is this configurable? Yes. The 'Package' option. Described in "Customizing the produced Debian

Gerry Reno wrote: source package (config options)" in the README.rst.

Ok, in my stdeb.cfg I put this: Package: myapp

And it generates the .deb without the 'python-' prefix. But, when I install this .deb it installs but no longer runs the postinst script. If I put back the 'python-' prefix then the postinst runs again. ????

Ok, I found it. In the debian/ control directory I had to rename the postinst script and remove the leading 'python-' from it as well. Regards, Gerry

Andrew, I see in the latest postings where setuptools is breaking with Python 2.6.3. Can you switch 'stdeb' over to using Distribute, which is being actively maintained? Regards, Gerry

OK I'm sorry for starting some kind of "battle" here :) I guess "sdist_*" stands for "source distribution", and likewise "bdist_*" stands for "binary distribution"? Tell a rookie like me: what is a binary distribution of a python application? It feels like an awkward concept. 2009/9/29 Gerry Reno :

Andrew Straw wrote:

...

Gerry Reno wrote:

...

Andrew Straw wrote:

...

Gerry Reno wrote:

...

What if stdeb only had the command 'bdist_deb' and had no other command.

I will not remove the "sdist_dsc" command from stdeb. I believe that the ability to produce debian source packages is much more important that the ability to produce binary packages which only target a single architecture and version of Debian/Ubuntu. "bdist_deb" is just a trivial add-on to that saving a couple lines of typing which may be helpful to people who aren't familiar with those lines. But I see the .dsc and associated files as much more useful as the compiled .deb files.

LMAO.  Dang it.  I am not suggesting or implying that you should remove the 'sdist_dsc' command.  I'm trying to illustrate that the commandA-opts are just that and nothing more: commandA-opts.  And whatever part of the build/install process that they apply to, then the commandA can use them for WHATEVER purpose.  Which means that since 'bdist_deb' calls 'sdist_dsc' it is perfectly ok to allow all 'sdist_dsc' options to be passed into the 'bdist_deb' command which can then pass them along to 'sdist_dsc'.  'bdist_deb' is a SUPERSET of 'sdist_dsc'.

I see -- you want to pass-through the options to sdist_dsc. Fine, implement this in a patch and I'll accept it.

Oh sure, I get to have all the fun.

...

...

...

For setting environment variables this way, they cannot be separated by a semicolon

The environment variables are not individually separated by a semicolon.  The WHOLE shell script environment variable declaration statement needs to be separated from the install line statement.

Simple demonstration of the problem with the 'echo' statement simulating the 'install' statement: $ a='123' b='456' echo test $a $b test $ a='123' b='456'; echo test $a $b test 123 456

Passing to a subprocess is different. We're passing to a subprocess:

$ a='123' python -c "import os; print os.environ['a']" 123 $ a='123'; python -c "import os; print os.environ['a']" Traceback (most recent call last):  File "<string>", line 1, in <module>  File "/usr/lib/python2.6/UserDict.py", line 22, in __getitem__    raise KeyError(key) KeyError: 'a'

Ok, I see what is happening.  In order for the env vars to be available to both the line itself AND to the subprocess it has to be declared as exported like this:

export a='123' b='456'; python -c "import os; print os.environ['a']" && echo $b 123 456

Now it will work both ways.

Regards, Gerry

_______________________________________________ Distutils-SIG maillist  -  Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

-- twitter.com/olofb olofb.wordpress.com olofb.wordpress.com/tag/english

2009/9/29 Gerry Reno :

It would be something like an .rpm or a .deb file or a windows installer file.

Or anything with a compiled c-extension. Or for that matter, pyc-files (but then you don't need it). -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64

2009/9/29 Ben Finney :

Olof Bjarnason writes:

...

I guess "sdist_*" stands for "source distribution", and likewise "bdist_*" stands for "binary distribution"?

Why are you guessing? You've been researching Python's distutils since several days ago, no? These terms are in the documentation you were directed to very early on:

   URL:http://docs.python.org/distutils/introduction.html

Your questions about these commands and terms suggest you haven't yet read the documentation that you acknowledged was a necessary first step.

Yeah sorry about that. I was going through a lot of different tools; and the clarification about bdist/sdist was not obvious from the first few pages of distutils documentation.

--  \         “Money is always to be found when men are to be sent to the |  `\   frontiers to be destroyed: when the object is to preserve them, | _o__)     it is no longer so.” —Voltaire, _Dictionnaire Philosophique_ | Ben Finney

_______________________________________________ Distutils-SIG maillist  -  Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

-- twitter.com/olofb olofb.wordpress.com olofb.wordpress.com/tag/english

2009/9/28 Olof Bjarnason :

2009/9/28 Andrew Straw :

...

Gerry Reno wrote:

...

Olof Bjarnason wrote:

...

...

Ok, the commands behave like makefile rules, once run they don't run again. But there are still several issues here:

Remember that I said that my goal with 'bdist_deb' was for users to have a SINGLE command to generate a .deb.

What needs to be achieved is for a command like this: $ python setup.py  bdist_deb OR $ python setup.py  bdist_deb --ignore-single-version-externally-managed

to be possible.  'bdist_deb' would call sdist_dsc internally with the necessary args WITHOUT having to explicitly put 'sdist_dsc' on the command line.

This would be _exactly_ what I'm looking for :)

I know.  I've had many of my users requesting exactly the same thing as well and I've been pursuing this for months now and when I got a 'bdist_deb' working with stdeb I knew I was getting close.  It just needs a little more tweeking and I think we'll have it.  Let's wait and see what Andrew says here. I don't understand what you're waiting on me for at this point.

Olaf -- as I understand it -- you need to work with Ubuntu 9.04. I just released stdeb 0.4 which supports this "bdist_deb" and doesn't pass the --single-version-externally-managed option anyway and therefore doesn't support the "--ignore-single-version-externally-managed" option. I don't understand why you need to pass "--ignore-single-version-externally-managed". "python setup.py bdist_deb" should work for you. Please report with specific information about expected and actual behavior if things are not working to your desire (taking care to read the, admittedly minimal, documentation in README.rst).

I tried "stdeb_run_setup", it got me a lot of files (and directories) although no .deb file.

Also tried "python setup.py bdist_deb" and ".. sdist_deb". Didn't work.

What is the command line I'm supposed to run in the setup.py directory?

OK I came a little further apt-get:ing debhelper and python-all-dev, then issuing sudo python -c "import stdeb; execfile('setup.py')" bdist_deb .. but it ended with this: Compatibility mode: using detected XS-Python-Version. dh_scrollkeeper dh_usrlocal dh_link dh_compress dh_fixperms dh_strip dh_makeshlibs dh_shlibdeps dh_installdeb dh_gencontrol dh_md5sums dh_builddeb varning, "debian/python-fortressdefender/DEBIAN/control" innehåller användardefinierat fält "Python-Version" dpkg-deb - fel: (upstream) version ("dev") innehåller inga siffror dpkg-deb: 1 fel i kontrollfilen dh_builddeb: command returned error code 512 make: *** [binary] Fel 1 dpkg-buildpackage: misslyckades: fakeroot debian/rules binary returnerade felkod 2 Traceback (most recent call last): File "<string>", line 1, in <module> File "setup.py", line 42, in <module> scripts = ["bin/fortressdefender"], File "/usr/lib/python2.6/distutils/core.py", line 152, in setup dist.run_commands() File "/usr/lib/python2.6/distutils/dist.py", line 975, in run_commands self.run_command(cmd) File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command cmd_obj.run() File "build/bdist.linux-x86_64/egg/stdeb/command/bdist_deb.py", line 40, in run File "build/bdist.linux-x86_64/egg/stdeb/util.py", line 73, in process_command File "build/bdist.linux-x86_64/egg/stdeb/util.py", line 31, in check_call stdeb.util.CalledProcessError: 2

Note: I'm a newbee on these tools, spent a total <1 hr so far on 'em, sorry for being so thick. It is a whole jungle to understand all these different tools/nomenclature and how it all hangs together. I'm new to distutils, setuptools, stdeb and .deb building in general.

...

Gerry -- I don't understand why you want to subvert the normal distutils way of doing things. Passing arguments to bdist_deb that are really arguments to sdist_dsc just isn't the way distutils does things. You're asking for the equivalent of being able to pass arguments to the distutils install command that are ultimately intended for the distutils build_ext command. If you want to test (and possibly implement) functionality such as adding a [sdist_dsc] section to setup.cfg where these options could just live so that you don't need to pass arguments at the command line, thats would be welcome.

Gerry, point 2 -- I still think you're not looking at your own big picture here -- AFAIK, you are bending over backwards to attempt to pass the "--ignore-single-version-externally-managed" so that you don't have to import setuptools in your setup.py file to avoid the monkeypatching that setuptools does. But, setuptools is explicitly imported in the debian/rules files created in stdeb 0.3.x. I get the impression that you're fixated on this argument passing business and not looking at the bigger picture of the architecture involved and whether your ultimate goal is even achievable with stdeb 0.3.x.

-Andrew

-- twitter.com/olofb olofb.wordpress.com olofb.wordpress.com/tag/english

-- twitter.com/olofb olofb.wordpress.com olofb.wordpress.com/tag/english

2009/9/28 Andrew Straw :

Olof Bjarnason wrote:

...

2009/9/28 Olof Bjarnason :

...

2009/9/28 Andrew Straw :

...

Gerry Reno wrote:

...

Olof Bjarnason wrote:

...

> Ok, the commands behave like makefile rules, once run they don't run > again. > But there are still several issues here: > > Remember that I said that my goal with 'bdist_deb' was for users to > have a > SINGLE command to generate a .deb. > > What needs to be achieved is for a command like this: > $ python setup.py  bdist_deb > OR > $ python setup.py  bdist_deb > --ignore-single-version-externally-managed > > to be possible.  'bdist_deb' would call sdist_dsc internally with the > necessary args WITHOUT having to explicitly put 'sdist_dsc' on the > command > line. > > This would be _exactly_ what I'm looking for :)

I know.  I've had many of my users requesting exactly the same thing as well and I've been pursuing this for months now and when I got a 'bdist_deb' working with stdeb I knew I was getting close.  It just needs a little more tweeking and I think we'll have it.  Let's wait and see what Andrew says here.

I don't understand what you're waiting on me for at this point.

Olaf -- as I understand it -- you need to work with Ubuntu 9.04. I just released stdeb 0.4 which supports this "bdist_deb" and doesn't pass the --single-version-externally-managed option anyway and therefore doesn't support the "--ignore-single-version-externally-managed" option. I don't understand why you need to pass "--ignore-single-version-externally-managed". "python setup.py bdist_deb" should work for you. Please report with specific information about expected and actual behavior if things are not working to your desire (taking care to read the, admittedly minimal, documentation in README.rst).

I tried "stdeb_run_setup", it got me a lot of files (and directories) although no .deb file.

Also tried "python setup.py bdist_deb" and ".. sdist_deb". Didn't work.

What is the command line I'm supposed to run in the setup.py directory?

OK I came a little further apt-get:ing debhelper and python-all-dev, then issuing

sudo python -c "import stdeb; execfile('setup.py')" bdist_deb

.. but it ended with this:

Compatibility mode: using detected XS-Python-Version.    dh_scrollkeeper    dh_usrlocal    dh_link    dh_compress    dh_fixperms    dh_strip    dh_makeshlibs    dh_shlibdeps    dh_installdeb    dh_gencontrol    dh_md5sums    dh_builddeb varning, "debian/python-fortressdefender/DEBIAN/control" innehåller användardefinierat fält "Python-Version" dpkg-deb - fel: (upstream) version ("dev") innehåller inga siffror dpkg-deb: 1 fel i kontrollfilen dh_builddeb: command returned error code 512 make: *** [binary] Fel 1 dpkg-buildpackage: misslyckades: fakeroot debian/rules binary returnerade felkod 2 Traceback (most recent call last):   File "<string>", line 1, in <module>   File "setup.py", line 42, in <module>     scripts = ["bin/fortressdefender"],   File "/usr/lib/python2.6/distutils/core.py", line 152, in setup     dist.run_commands()   File "/usr/lib/python2.6/distutils/dist.py", line 975, in run_commands     self.run_command(cmd)   File "/usr/lib/python2.6/distutils/dist.py", line 995, in run_command     cmd_obj.run()   File "build/bdist.linux-x86_64/egg/stdeb/command/bdist_deb.py", line 40, in run   File "build/bdist.linux-x86_64/egg/stdeb/util.py", line 73, in process_command   File "build/bdist.linux-x86_64/egg/stdeb/util.py", line 31, in check_call stdeb.util.CalledProcessError: 2

please do:

sudo apt-get install build-essential sudo apt-get install fakeroot

Did both, added no package to my system. (I guess you are intending to supply a stdeb.deb in the future to fix all dependencies ;) ? It would be slick with a script that did this command line: sudo python -c "import stdeb; execfile('setup.py')" bdist_deb) I'm going to bed over here. Attaching the whole dump of the above command line.

And try again.

-Andrew

-- twitter.com/olofb olofb.wordpress.com olofb.wordpress.com/tag/english