bdist_rpm fails due to bytecompiled modules
hello, i have a very basic pure-python package and script that i would like to distribute as a noarch rpm using distutils. i am using FC5 as my development platform and have prepared a setup.py that properly builds and installs my application. the first problem i have is that the bdist_rpm command results in an %install section as follows: %install python setup.py install --single-version-externally-managed --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES the trouble here with that %install section is that the python modules get bytecompiled and thus turned into native code. i'd prefer for that not to happen, so i have written my own install.sh script and i provide that to my python setup.py call as follows: $ python setup.py bdist_rpm --spec-only --install-script /path/to/install.sh my install.sh passes the --optimize 0 and --no-compile arguments to distutils, resulting in no .pyc or .pyo files being installed in my RPM_BUILD_ROOT. the contents of my install.sh are as follows: python setup.py install --single-version-externally-managed --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES --optimize 0 --no-compile i thought that would result in no byte compiled files but unfortunately i was wrong. i'm not sure what's causing it, but rpmbuild ends up calling brp-python-bytecompile to compile the .py files. i noticed in the ChangeLog for rpm the following: - force *.py->*.pyo byte code compilation with brp-python-bytecompile. so it appears to me as tho there is no way to assemble a noarch RPM using distutils. does that sound accurate to you all? or are those folks out there using distutils and an old version of rpm (prior to 2005) that are able to assemble noarch RPMs? -c -- WeatherNet Observations for station: home Temperature: 54.80F Pressure: 30.02in; Dew Point: 30.76F (40%) Wind: 248 at 5 mph Recorded: 17:54:59 10/25/06 (http://wsdl.wxnet.org/inquiry/binding.wsdl)
On Oct 25, 2006, at 8:05 PM, Christopher Blunck wrote:
the trouble here with that %install section is that the python modules get bytecompiled and thus turned into native code. i'd prefer for that not to happen, so i have written my own install.sh script and i provide that to my python setup.py call as follows:
.pyc and .pyo files aren't native code, they are bytecode files that work on all platforms. There should therefore be no need to surpress bytecode compilation. Ronald
On Wed, Oct 25, 2006 at 09:01:03PM +0200, Ronald Oussoren wrote:
On Oct 25, 2006, at 8:05 PM, Christopher Blunck wrote:
the trouble here with that %install section is that the python modules get bytecompiled and thus turned into native code. i'd prefer for that not to happen, so i have written my own install.sh script and i provide that to my python setup.py call as follows:
.pyc and .pyo files aren't native code, they are bytecode files that work on all platforms. There should therefore be no need to surpress bytecode compilation.
oh i see... my mistake. :) how do you workaround the "installed (but unpackaged) file(s) found" error related to the .pyo/.pyc files? i noticed it's a python bug as well as a redhat bug. ps - the original reason i went down the rabbit hole of --optimization 0 and --no-compile was that i was trying to get around the installed but unpackaged files error. my thought was that if i could prevent the creation of the .pyc and .pyo files then i could avoid the rpmbuild error. -c -- WeatherNet Observations for station: home Temperature: 54.20F Pressure: 30.02in; Dew Point: 33.22F (45%) Wind: 64 at 3 mph Recorded: 19:04:50 10/25/06 (http://wsdl.wxnet.org/inquiry/binding.wsdl)
On 10/25/06, Christopher Blunck <chris@wxnet.org> wrote:
how do you workaround the "installed (but unpackaged) file(s) found" error related to the .pyo/.pyc files? i noticed it's a python bug as well as a redhat bug.
Not sure what you call a bug here. The creation of .pyc and .pyo files is an important stage and shouldn't be omitted when building RPMs, nor should the files be skipped (deleted and not included in %files). This way stuff like SELinux permissions get set correctly when the RPM is installed. Cheers, -- Konstantin Ryabitsev Montréal, Québec
On Wed, Oct 25, 2006 at 05:15:32PM -0400, Konstantin Ryabitsev wrote:
On 10/25/06, Christopher Blunck <chris@wxnet.org> wrote:
how do you workaround the "installed (but unpackaged) file(s) found" error related to the .pyo/.pyc files? i noticed it's a python bug as well as a redhat bug.
Not sure what you call a bug here. The creation of .pyc and .pyo files is an important stage and shouldn't be omitted when building RPMs, nor should the files be skipped (deleted and not included in %files). This way stuff like SELinux permissions get set correctly when the RPM is installed.
distutils/setuptools creates .pyc files - that's ok. here's how the problem occurs: 1.) distutils generates a .spec file with an %install that calls setup.py like below. .py and .pyc files are generated, and are included in the INSTALLED_FILES record. see below for what is called in %install: python setup.py install --single-version-externally-managed --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES 2.) rpmbuild is called, and it generates a shell script that calls the command above. AFTER that command is executed, rpmbuild calls some strip commands as well as brp-python-bytecompile. brp-python-bytecompile generates .pyo files from .pyc but it does not update INSTALLED_FILES (because that is a distutils artifact that rpmbuild doesn't know about) 3.) rpmbuild calls into %install, and INSTALLED_FILES is loaded. it does not include the .pyo files that rpmbuild created. since there are unpackaged files (the .pyo files) that exist in the installation directory that are not referenced in the INSTALLED_FILES file, rpmbuild chokes with an error. step #3 is where the error occurs. it's basically an inconsistency between distutils and rpmbuild. i can run distutils to generate optimized modules (.pyo) but that doesn't apply to my scripts. as a result i still have unpackaged files... -c -- WeatherNet Observations for station: home Temperature: 51.60F Pressure: 30.04in; Dew Point: 33.02F (49%) Wind: 81 at 0 mph Recorded: 21:24:56 10/25/06 (http://wsdl.wxnet.org/inquiry/binding.wsdl)
On 10/25/06, Christopher Blunck <chris@wxnet.org> wrote:
Not sure what you call a bug here. The creation of .pyc and .pyo files is an important stage and shouldn't be omitted when building RPMs, nor should the files be skipped (deleted and not included in %files). This way stuff like SELinux permissions get set correctly when the RPM is installed.
distutils/setuptools creates .pyc files - that's ok.
here's how the problem occurs:
1.) distutils generates a .spec file with an %install that calls setup.py like below. .py and .pyc files are generated, and are included in the INSTALLED_FILES record. see below for what is called in %install:
python setup.py install --single-version-externally-managed --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES
2.) rpmbuild is called, and it generates a shell script that calls the command above. AFTER that command is executed, rpmbuild calls some strip commands as well as brp-python-bytecompile. brp-python-bytecompile generates .pyo files from .pyc but it does not update INSTALLED_FILES (because that is a distutils artifact that rpmbuild doesn't know about)
3.) rpmbuild calls into %install, and INSTALLED_FILES is loaded. it does not include the .pyo files that rpmbuild created. since there are unpackaged files (the .pyo files) that exist in the installation directory that are not referenced in the INSTALLED_FILES file, rpmbuild chokes with an error.
step #3 is where the error occurs. it's basically an inconsistency between distutils and rpmbuild. i can run distutils to generate optimized modules (.pyo) but that doesn't apply to my scripts. as a result i still have unpackaged files...
Yes, it looks like distutils is not anticipating the new behaviour from rpmbuild (both .pyc and .pyo will be compiled by brp-python-bytecompile). It's probably something that should be fixed when generating the %files section. Regards, -- Konstantin Ryabitsev Montréal, Québec
Christopher Blunck wrote:
distutils/setuptools creates .pyc files - that's ok.
here's how the problem occurs:
1.) distutils generates a .spec file with an %install that calls setup.py like below. .py and .pyc files are generated, and are included in the INSTALLED_FILES record. see below for what is called in %install:
python setup.py install --single-version-externally-managed --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES
2.) rpmbuild is called, and it generates a shell script that calls the command above. AFTER that command is executed, rpmbuild calls some strip commands as well as brp-python-bytecompile. brp-python-bytecompile generates .pyo files from .pyc but it does not update INSTALLED_FILES (because that is a distutils artifact that rpmbuild doesn't know about)
3.) rpmbuild calls into %install, and INSTALLED_FILES is loaded. it does not include the .pyo files that rpmbuild created. since there are unpackaged files (the .pyo files) that exist in the installation directory that are not referenced in the INSTALLED_FILES file, rpmbuild chokes with an error.
step #3 is where the error occurs. it's basically an inconsistency between distutils and rpmbuild. i can run distutils to generate optimized modules (.pyo) but that doesn't apply to my scripts. as a result i still have unpackaged files...
Thanks for the detailed analysis ! It seems a workaround is to provide an install command that creates those .pyo files upfront, so they find their way into the INSTALLED_FILES list. Thus, I tried 'python setup.py bdist_rpm --install-script=install', where the content of 'install' is 'python setup.py install --optimize=1 --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES', and voila ! It works ! Not sure whether it is worth holding your breath for someone to fix distutils. :-( Thanks, Stefan -- ...ich hab' noch einen Koffer in Berlin...
At 11:28 PM 12/5/2006 -0500, Stefan Seefeld wrote:
Thus, I tried 'python setup.py bdist_rpm --install-script=install', where the content of 'install' is 'python setup.py install --optimize=1 --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES', and voila ! It works !
A simpler fix is to put: [install] optimize=1 in the project's setup.cfg, or perhaps better yet, in ~/.pydistutils.cfg or in $prefix/lib/python2.x/distutils/distutils.cfg. This will then always set --optimize=1 when install is invoked.
Phillip J. Eby wrote:
At 11:28 PM 12/5/2006 -0500, Stefan Seefeld wrote:
Thus, I tried 'python setup.py bdist_rpm --install-script=install', where the content of 'install' is 'python setup.py install --optimize=1 --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES', and voila ! It works !
A simpler fix is to put:
[install] optimize=1
in the project's setup.cfg, or perhaps better yet, in ~/.pydistutils.cfg or in $prefix/lib/python2.x/distutils/distutils.cfg. This will then always set --optimize=1 when install is invoked.
I was thinking about that, but couldn't figure out whether the project's setup.cfg would be used when 'python setup.py install' is invoked from within rpmbuild. The docs are silent about this (and other) precious detail(s). Thanks for the tip ! Stefan -- ...ich hab' noch einen Koffer in Berlin...
participants (5)
-
Christopher Blunck -
Konstantin Ryabitsev -
Phillip J. Eby -
Ronald Oussoren -
Stefan Seefeld