wheel building help needed
Hi, So, i'm trying to get around building the wheel using pip wheel and I'm having some troubles with it. I would appreciate if you could clarify some of the questions for me: I have a structure like that: ./ wheel/ __main__.py my/__init__.py cool/__init__.py package/prog.py __init__.py MANIFEST.in setup.py setup.cfg Where setup.cfg has only [wheel] universal=1 Manifest has: include wheel/__main__.py recursive_include wheel *.py I'm issuing "pip wheel ." and it does put a wheel with some generated info in it into the ./wheelhouse, but there are issues: 1. No python files added, neither __main__.py nor prog.py make it to the wheel, so includes are not working 2. Despite the fact that I have universal=1 there is no py2.py3 in the name it says cp33-cp33m Could anybody hint anything? In addition to that I failed to find the rules for manifest and setup.cfg files (acceptable sections, values, syntax) so i would appreciate if anybody could share a link to the stuff. Thanks a lot! Eugene
On 5 February 2014 22:20, Evgeny Sazhin <eugene@sazhin.us> wrote:
1. No python files added, neither __main__.py nor prog.py make it to the wheel, so includes are not working
The manifest is for sdists, not wheels. So includes won't "work" in the sense that you seem to be expecting. You need to specify your package details in setup.py - you don't say if you did. I suspect you're not getting a universal wheel because it looks like it isn't pure Python - maybe because it doesn't have the right files in it, I can't tell from what you're showing. You still seem to be making wheels with the intention that they are executable. I apologise if I'm reading too much into your directory structure, but if that's what you are trying to do I can well imagine that you'll get odd results as you're trying to do something the system wasn't designed for. Paul
On Wed, Feb 5, 2014 at 6:01 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 5 February 2014 22:20, Evgeny Sazhin <eugene@sazhin.us> wrote:
1. No python files added, neither __main__.py nor prog.py make it to the wheel, so includes are not working
The manifest is for sdists, not wheels. So includes won't "work" in the sense that you seem to be expecting. You need to specify your package details in setup.py - you don't say if you did. I suspect you're not getting a universal wheel because it looks like it isn't pure Python - maybe because it doesn't have the right files in it, I can't tell from what you're showing.
I got rid of the wheel folder as the machinery behind the wheel building (or between keyboard and chair) is not smart enough to understand how to work with packages that are one level down from the root. my setup.py is : setup( name="prog", version="0.1", #etc, packages=find_packages() ) I got the wheel properly built using python setup.py bdist_wheel (i was able to do it before, but didn't realize the manifest stuff is for that way of building, not for the pip wheel). So "pip wheel" usage remains to be unclear...
You still seem to be making wheels with the intention that they are executable. I apologise if I'm reading too much into your directory structure, but if that's what you are trying to do I can well imagine that you'll get odd results as you're trying to do something the system wasn't designed for.
Paul
Yes, not only i'm making wheel executable, but i also will most probably build around the fact that it is zip-readable and therefore I will add them to the PYTHONPATH to run, just like java classpath and jar (I know - stubborn me) - that's the only thing that makes any sense! After all those arguments about C extensions and whatnot i still think that enforcing the unzipping and deploying stuff is not what packaging should provide as a solution. Plus as you see I'm trying to build universal wheel - pure python. Nobody so far could prove or at least explain why it is bad idea *for pure python*, so I think this is the way to go. Thanks, Eugene
You are probably just missing the package_dir argument to setup(), adequately documented at http://docs.python.org/2/distutils/examples.html Pass package_dir={'': 'src'}, to indicate that the root module starts in 'src'. MANIFEST.in is needed. It is always a bit awkward to write. On Wed, Feb 5, 2014 at 6:41 PM, Evgeny Sazhin <eugene@sazhin.us> wrote:
On Wed, Feb 5, 2014 at 6:01 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 5 February 2014 22:20, Evgeny Sazhin <eugene@sazhin.us> wrote:
1. No python files added, neither __main__.py nor prog.py make it to the wheel, so includes are not working
The manifest is for sdists, not wheels. So includes won't "work" in the sense that you seem to be expecting. You need to specify your package details in setup.py - you don't say if you did. I suspect you're not getting a universal wheel because it looks like it isn't pure Python - maybe because it doesn't have the right files in it, I can't tell from what you're showing.
I got rid of the wheel folder as the machinery behind the wheel building (or between keyboard and chair) is not smart enough to understand how to work with packages that are one level down from the root.
my setup.py is :
setup( name="prog", version="0.1", #etc, packages=find_packages()
)
I got the wheel properly built using python setup.py bdist_wheel (i was able to do it before, but didn't realize the manifest stuff is for that way of building, not for the pip wheel). So "pip wheel" usage remains to be unclear...
You still seem to be making wheels with the intention that they are executable. I apologise if I'm reading too much into your directory structure, but if that's what you are trying to do I can well imagine that you'll get odd results as you're trying to do something the system wasn't designed for.
Paul
Yes, not only i'm making wheel executable, but i also will most probably build around the fact that it is zip-readable and therefore I will add them to the PYTHONPATH to run, just like java classpath and jar (I know - stubborn me) - that's the only thing that makes any sense! After all those arguments about C extensions and whatnot i still think that enforcing the unzipping and deploying stuff is not what packaging should provide as a solution. Plus as you see I'm trying to build universal wheel - pure python. Nobody so far could prove or at least explain why it is bad idea *for pure python*, so I think this is the way to go.
Thanks, Eugene _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Feb 5, 2014, at 9:10 PM, Daniel Holth <dholth@gmail.com> wrote:
You are probably just missing the package_dir argument to setup(), adequately documented at http://docs.python.org/2/distutils/examples.html
Pass
package_dir={'': 'src'},
to indicate that the root module starts in 'src'.
MANIFEST.in is needed. It is always a bit awkward to write.
Thank you Daniel! I will definitely take a look at that. I have to say though that this is one of the problems i have with the whole python packaging: It is scattered all over the place. I’m importing setuptools in the setup.py to build wheel while i obviously miss some information about distutils? I think it could be very beneficial to have it all moved under one umbrella project or wrapper or something like that. I as a user am interested in centralized thing that knows how to take care of everything i need in order to make a wheel. Why not to have only setup.cfg (even better wheel.cfg) that describes all necessary things including includes and package details and then have “wheel" command that knows to create it properly basing on this config? One config file, one dependencies file (requirements.txt) - one command that does the work = what the doctor ordered! No? Thanks again, Eugene
On Wed, Feb 5, 2014 at 6:41 PM, Evgeny Sazhin <eugene@sazhin.us> wrote:
On Wed, Feb 5, 2014 at 6:01 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 5 February 2014 22:20, Evgeny Sazhin <eugene@sazhin.us> wrote:
1. No python files added, neither __main__.py nor prog.py make it to the wheel, so includes are not working
The manifest is for sdists, not wheels. So includes won't "work" in the sense that you seem to be expecting. You need to specify your package details in setup.py - you don't say if you did. I suspect you're not getting a universal wheel because it looks like it isn't pure Python - maybe because it doesn't have the right files in it, I can't tell from what you're showing.
I got rid of the wheel folder as the machinery behind the wheel building (or between keyboard and chair) is not smart enough to understand how to work with packages that are one level down from the root.
my setup.py is :
setup( name="prog", version="0.1", #etc, packages=find_packages()
)
I got the wheel properly built using python setup.py bdist_wheel (i was able to do it before, but didn't realize the manifest stuff is for that way of building, not for the pip wheel). So "pip wheel" usage remains to be unclear...
You still seem to be making wheels with the intention that they are executable. I apologise if I'm reading too much into your directory structure, but if that's what you are trying to do I can well imagine that you'll get odd results as you're trying to do something the system wasn't designed for.
Paul
Yes, not only i'm making wheel executable, but i also will most probably build around the fact that it is zip-readable and therefore I will add them to the PYTHONPATH to run, just like java classpath and jar (I know - stubborn me) - that's the only thing that makes any sense! After all those arguments about C extensions and whatnot i still think that enforcing the unzipping and deploying stuff is not what packaging should provide as a solution. Plus as you see I'm trying to build universal wheel - pure python. Nobody so far could prove or at least explain why it is bad idea *for pure python*, so I think this is the way to go.
Thanks, Eugene _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 6 February 2014 05:37, Evgeny Sazhin <eugene@sazhin.us> wrote:
I think it could be very beneficial to have it all moved under one umbrella project or wrapper or something like that. I as a user am interested in centralized thing that knows how to take care of everything i need in order to make a wheel. Why not to have only setup.cfg (even better wheel.cfg) that describes all necessary things including includes and package details and then have "wheel" command that knows to create it properly basing on this config? One config file, one dependencies file (requirements.txt) - one command that does the work = what the doctor ordered! No?
This is one of the major things being worked on at the moment - initially, putting the documentation together in one place (http://python-packaging-user-guide.readthedocs.org/en/latest/) and eventually having a more "unified" build process, but that's some way off. Experience and feedback like yours is immensely useful in making sure things are documented in a way that is accessible to newcomers (most of us round here are a bit too close to the problem to know what's needed :-)). So many thanks for taking the time to post your experiences. Paul
On Thu, Feb 6, 2014 at 12:37 AM, Evgeny Sazhin <eugene@sazhin.us> wrote:
On Feb 5, 2014, at 9:10 PM, Daniel Holth <dholth@gmail.com> wrote:
You are probably just missing the package_dir argument to setup(), adequately documented at http://docs.python.org/2/distutils/examples.html
Pass
package_dir={'': 'src'},
to indicate that the root module starts in 'src'.
MANIFEST.in is needed. It is always a bit awkward to write.
Thank you Daniel! I will definitely take a look at that.
I have to say though that this is one of the problems i have with the whole python packaging: It is scattered all over the place. I'm importing setuptools in the setup.py to build wheel while i obviously miss some information about distutils?
I think it could be very beneficial to have it all moved under one umbrella project or wrapper or something like that. I as a user am interested in centralized thing that knows how to take care of everything i need in order to make a wheel. Why not to have only setup.cfg (even better wheel.cfg) that describes all necessary things including includes and package details and then have "wheel" command that knows to create it properly basing on this config? One config file, one dependencies file (requirements.txt) - one command that does the work = what the doctor ordered! No?
Thanks again, Eugene
I would like to see more people experiment with distutils replacements. Bento and distil do that to some extent. We would like to make it very easy to switch out the setup.py implementation for something else.
On Wed, Feb 05, 2014 at 09:10:31PM -0500, Daniel Holth wrote:
MANIFEST.in is needed. It is always a bit awkward to write.
https://pypi.python.org/pypi/check-manifest is here to help. Marius Gedminas -- Real programmers don't write in BASIC. Actually, no programmers write in BASIC after reaching puberty.
On Wed, Feb 5, 2014 at 9:10 PM, Daniel Holth <dholth@gmail.com> wrote:
You are probably just missing the package_dir argument to setup(), adequately documented at http://docs.python.org/2/distutils/examples.html
Pass
package_dir={'': 'src'},
to indicate that the root module starts in 'src'.
MANIFEST.in is needed. It is always a bit awkward to write.
I tried this and it did not work. It keeps missing the python files. The only way it worked and created the expected wheel ( at least the python files are there from the package and the universal=1 resulted in correct tag) was when the package dir was directly in the root. I also noticed another problem that in manifest include . *.py doesn't work, neither does the include * So my __main__.py now located in the root doesn't get inside the wheel. So you said PEX is worth checking, huh?...;) Thanks, Eugene
On Wed, Feb 5, 2014 at 6:41 PM, Evgeny Sazhin <eugene@sazhin.us> wrote:
On Wed, Feb 5, 2014 at 6:01 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 5 February 2014 22:20, Evgeny Sazhin <eugene@sazhin.us> wrote:
1. No python files added, neither __main__.py nor prog.py make it to the wheel, so includes are not working
The manifest is for sdists, not wheels. So includes won't "work" in the sense that you seem to be expecting. You need to specify your package details in setup.py - you don't say if you did. I suspect you're not getting a universal wheel because it looks like it isn't pure Python - maybe because it doesn't have the right files in it, I can't tell from what you're showing.
I got rid of the wheel folder as the machinery behind the wheel building (or between keyboard and chair) is not smart enough to understand how to work with packages that are one level down from the root.
my setup.py is :
setup( name="prog", version="0.1", #etc, packages=find_packages()
)
I got the wheel properly built using python setup.py bdist_wheel (i was able to do it before, but didn't realize the manifest stuff is for that way of building, not for the pip wheel). So "pip wheel" usage remains to be unclear...
You still seem to be making wheels with the intention that they are executable. I apologise if I'm reading too much into your directory structure, but if that's what you are trying to do I can well imagine that you'll get odd results as you're trying to do something the system wasn't designed for.
Paul
Yes, not only i'm making wheel executable, but i also will most probably build around the fact that it is zip-readable and therefore I will add them to the PYTHONPATH to run, just like java classpath and jar (I know - stubborn me) - that's the only thing that makes any sense! After all those arguments about C extensions and whatnot i still think that enforcing the unzipping and deploying stuff is not what packaging should provide as a solution. Plus as you see I'm trying to build universal wheel - pure python. Nobody so far could prove or at least explain why it is bad idea *for pure python*, so I think this is the way to go.
Thanks, Eugene _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Evgeny, while you can probably make wheels do what you want, if you're interested in single file executables, you're almost certainly better off using one of the tools designed to make those easier to work with (like Twitter's recently discussed PEX format, or PEP 441). As you have discovered, cross platform single file executables are an area where the currently published documentation is even less polished and coherent than that for the rest of the packaging infrastructure :) Cheers, Nick.
On Thu, Feb 6, 2014 at 4:54 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Evgeny, while you can probably make wheels do what you want, if you're interested in single file executables, you're almost certainly better off using one of the tools designed to make those easier to work with (like Twitter's recently discussed PEX format, or PEP 441).
As you have discovered, cross platform single file executables are an area where the currently published documentation is even less polished and coherent than that for the rest of the packaging infrastructure :)
Cheers, Nick.
Nick, Thank you for pointing that out. I took a brief look at the pyz stuff and while it might be good generally - it doesn't help me much. There are two problems i'm trying to solve : dependency management and jar-like behavior (as libraries or executables). Pip + wheel combo seems to be geared better for that task (may be as you guys are pointing out, not intentionally, but still seems to be better approach). We will definitely take a look at PEX as from the very scattered info i got so far they seem to be very good at dependency management - so that might be an option. Thanks, Eugene
Happy to answer any specific questions about PEX here or off-thread. I'm hacking on wheel support in my spare time -- hope to ship that sometime in the next couple weeks. The other big upcoming change is standardizing docstrings throughout so that we can get a proper API published on readthedocs. ~brian On Thu, Feb 6, 2014 at 2:00 PM, Evgeny Sazhin <eugene@sazhin.us> wrote:
On Thu, Feb 6, 2014 at 4:54 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Evgeny, while you can probably make wheels do what you want, if you're interested in single file executables, you're almost certainly better off using one of the tools designed to make those easier to work with (like Twitter's recently discussed PEX format, or PEP 441).
As you have discovered, cross platform single file executables are an area where the currently published documentation is even less polished and coherent than that for the rest of the packaging infrastructure :)
Cheers, Nick.
Nick,
Thank you for pointing that out. I took a brief look at the pyz stuff and while it might be good generally - it doesn't help me much. There are two problems i'm trying to solve : dependency management and jar-like behavior (as libraries or executables). Pip + wheel combo seems to be geared better for that task (may be as you guys are pointing out, not intentionally, but still seems to be better approach).
We will definitely take a look at PEX as from the very scattered info i got so far they seem to be very good at dependency management - so that might be an option.
Thanks, Eugene _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Brian, that's very kind of you - i will definitely ping you for questions, but after my small vacation;) Thanks, Eugene On Thu, Feb 6, 2014 at 5:23 PM, Brian Wickman <wickman@gmail.com> wrote:
Happy to answer any specific questions about PEX here or off-thread. I'm hacking on wheel support in my spare time -- hope to ship that sometime in the next couple weeks. The other big upcoming change is standardizing docstrings throughout so that we can get a proper API published on readthedocs.
~brian
On Thu, Feb 6, 2014 at 2:00 PM, Evgeny Sazhin <eugene@sazhin.us> wrote:
On Thu, Feb 6, 2014 at 4:54 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Evgeny, while you can probably make wheels do what you want, if you're interested in single file executables, you're almost certainly better off using one of the tools designed to make those easier to work with (like Twitter's recently discussed PEX format, or PEP 441).
As you have discovered, cross platform single file executables are an area where the currently published documentation is even less polished and coherent than that for the rest of the packaging infrastructure :)
Cheers, Nick.
Nick,
Thank you for pointing that out. I took a brief look at the pyz stuff and while it might be good generally - it doesn't help me much. There are two problems i'm trying to solve : dependency management and jar-like behavior (as libraries or executables). Pip + wheel combo seems to be geared better for that task (may be as you guys are pointing out, not intentionally, but still seems to be better approach).
We will definitely take a look at PEX as from the very scattered info i got so far they seem to be very good at dependency management - so that might be an option.
Thanks, Eugene
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
participants (6)
-
Brian Wickman -
Daniel Holth -
Evgeny Sazhin -
Marius Gedminas -
Nick Coghlan -
Paul Moore