wheels on sys.path clarification (reboot)
Eh, I think both 1 and 3 are things that are possibly reasonable to happen and they are both things that I've contemplated as things to bring forward in using xz as an alternative compression format. Even if #1 would need a major revision of Wheel to happen adding official "support" for zip import means that the change would have to be weighed against also breaking that backwards compatibility.
Eh, please don’t break it!! Improve! Keep the ability to add wheels to python path. I don’t care how, zip import, non-zip import, whatever - allow for jar-like behavior - people will thank you if it will work properly !;) I know, i’m probably alone in the desert…l;)
On Jan 30, 2014, at 12:33 AM, Evgeny Sazhin <eugene@sazhin.us> wrote:
Eh, I think both 1 and 3 are things that are possibly reasonable to happen and they are both things that I've contemplated as things to bring forward in using xz as an alternative compression format. Even if #1 would need a major revision of Wheel to happen adding official "support" for zip import means that the change would have to be weighed against also breaking that backwards compatibility.
Eh, please don’t break it!! Improve! Keep the ability to add wheels to python path. I don’t care how, zip import, non-zip import, whatever - allow for jar-like behavior - people will thank you if it will work properly !;) I know, i’m probably alone in the desert…l;)
OTOH if this is such a bad idea - what would be the least intrusive way for me to deploy the code? What about unzipped wheel? If I have wheels in one location and then my release procedure knows to unzip them in the folder that matches the wheel name and structure completely, would it be possible to add this unzipped folder to the PYTHONPATH and run the program? Is that closer to what you guys are preaching?? Thanks, Eugene
On 30 January 2014 05:33, Evgeny Sazhin <eugene@sazhin.us> wrote:
Eh, I think both 1 and 3 are things that are possibly reasonable to happen and they are both things that I've contemplated as things to bring forward in using xz as an alternative compression format. Even if #1 would need a major revision of Wheel to happen adding official "support" for zip import means that the change would have to be weighed against also breaking that backwards compatibility.
Eh, please don't break it!! Improve! Keep the ability to add wheels to python path. I don't care how, zip import, non-zip import, whatever - allow for jar-like behavior - people will thank you if it will work properly !;) I know, i'm probably alone in the desert...l;)
This is the biggest concern I see with "promoting" wheels being directly importable via zipimport (I say "promoting" and not "specifying" deliberately, but I don't want to get back into the process debate). People like Evgeny (no offense intended here, but your post is a good example to use) coming from a Java background will see importable wheels and *think* that they are similar in purpose to jars. This is not, and never will be, true. Python is not Java, deployment by putting jar-like files on sys.path is not an advisable workflow for Python - no matter what parallels with Java might make it look attractive. Pointing out the implication that wheels can be put on sys.path in specialised circumstances is entirely reasonable when talking to a Python audience, but it triggers entirely the wrong associations when a person with a Java background sees the statement. Evgeny - for your purposes, you'd be better advised to see wheels as similar in principle to Windows MSI installers (they aren't, but it will avoid triggering incorrect analogies). I don't have good references or pointers to good Python development or deployment practices, but you may want to ask around on python-list. (Be careful, you may hear suggestions that you use eggs because they are similar to jars - while that is true, the way Python works means that the experience with eggs is far from being as natural as the jar experience you're used to). Paul
-------------------------------------------- On Thu, 30/1/14, Paul Moore <p.f.moore@gmail.com> wrote:
This is the biggest concern I see with "promoting" wheels being directly importable via zipimport (I say "promoting" and not "specifying" deliberately, but I don't want to get back into the process debate).
In my view, we can have our cake and eat it. Those who don't want their wheels to be mountable can say so in their wheel metadata. I expect that distlib will honour such metadata (once the details have been worked out) and not mount wheels which aren't supposed to be mounted. However, it's perfectly easy to write code that runs from zip, as long as you know that's a deployment option you want to support. So it doesn't make sense to prevent that useful functionality - even pip is using it, I believe ;-) - just because some people think it's a bad idea, for reasons that are hardly compelling. To say that we should keep packaging separate from importing is in some sense a religious argument, unless sound technical reasons are given for it. Java proves otherwise otherwise - and for those who hate Java, that's a religious viewpoint, too ... The argument that importing wheels will cause problems is trumped by allowing the mountability to be configurable by wheel metadata: the builder of a wheel, by saying it is mountable, is agreeing to all that entails in terms of handling package resources appropriately, etc. So the support burden shifts to them, and not to wheel, distlib or any packaging tool. So much heat over process, but so little light over exactly why *appropriately designed* software deployed in wheels shouldn't be importable. No detailed analysis of any problem with wheels taking into account the differences from eggs (the things Daniel mentioned in the other thread "problems with eggs?", plus the fact that there's no sys.path manipulation *magic*). Just blanket statements to the effect that "eggs were importable and bad, so wheels must be too" smacks of superstition, not engineering. I would like the detractors of importable wheels to put their technical reasoning on the table, not use process debates to try to revert situations they're not happy with. That technical reasoning should address wheels as they are now and avoid referring to eggs as the bogeyman. The only reason I've heard from detractors so far is that software not designed to run from zip can give unexpected results when run from zip, which could give the wheel format a bad name. Given that a wheel publisher can have a say on importability, this issue seems to me to be adequately addressed. No detractor has come up with any other solid reasoning as to why the useful feature of wheel importability is bad. I invite them to put up, or ... we'll still be in the dark the next time this debate comes around ;-) Regards, Vinay Sajip
On 30 January 2014 09:45, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
So much heat over process, but so little light over exactly why *appropriately designed* software deployed in wheels shouldn't be importable.
Good point. In my view, *appropriately designed* software is fine, and a metadata flag is a good idea. Although obviously that makes "mounting" a little more complex (check the flag) and argues for a mount method rather than people just fiddling sys.path. There's a lot of leeway for dispute over "appropriate design" and who gets to say, but that's not a technical issue. My one technical issue is with going beyond zipimport behaviour to the point of extracting DLLs to the filesystem. I remain -1 on that feature, and I believe I have explained why I think there are issues (and why I think that any solution should be part of zipimport and not added on in library or user code). But I'm happy to go through the details again, if you like - or just to accept that I don't need to use the feature. (Would you be willing to add some sort of "never extract C extensions regardless of what the metadata might say" option to wheel mount? It's not critical, as mounting random wheels without knowing how they work is clearly bad, but it does add a level of assurance that might be helpful,) The remaining issues seem to be mainly of education and expectations - people coming from non-Python (particularly Java) backgrounds have a set of assumptions that aren't easy to translate into Python terms. That's definitely not a technical problem. Paul.
My one technical issue is with going beyond zipimport behaviour to the point of extracting DLLs to the filesystem. I remain -1 on that feature, and I believe I have explained why I think there are issues (and why I think that any solution should be part of zipimport and not added on in library or user code). But I'm happy to go through the details again, if you like - or just to accept that I don't
Yes please, let's get into some details. Of course I understand that you might not want to use the feature, but I don't understand the -1 on the feature per se - whether it is in distlib or in zipimport is a secondary consideration. I agree that zipimport is the logical place for it, but ISTM the reason why it can't go in there just yet is also the reason why one might have some reservations about the feature: binary compatibility. I accept that this not yet a fully resolved issue in general (cf. the parallel discussion about numpy), but if we can isolate these issues, we can perhaps tackle them. But for me, that's the main reason why this part of the distlib API is experimental. Since zipimport makes no prescription about the contents of a zip (beyond describing how importing works), there is no agreed place to place metadata information such as about binary compatibility. Nor is binary compatibility between third-party packages of core concern to python-dev, so I'm not sure discussions there will be fruitful. However, such compatibility is a valid concern here, so I would expect useful input to come from interested parties. Also, the wheel format already caters for a limited set of binary compatibility info to be communicated, but this information is incomplete, which results in reservations about the dangers of extracting C extensions. In a sense, the contentiousness of extraction of these from a wheel is a red herring, because the exact same issues would arise if you installed from the wheel and then tried to use software which purported to be binary compatible with your system, but wasn't. That's why we don't put Linux wheels on PyPI, right? It's to avoid binary wheels compiled on e.g. CentOS ending up on an e.g. Ubuntu system which is not quite binary compatible. But that doesn't solve the problem at source, so much as act as a prophylactic. Much of the Python community works in a POSIX environment where build- from-source is the norm and binary compatibility is a non-issue. In a corporate environment with a homogeneous infrastructure, the same applies; but in a heterogeneous environment, not so much. I think some benefit would accrue to packaging as a whole if we had better definitions of binary compatibility, the ability to express it in more detail, how to test for it at installation time and so on. I think this is one of the areas where we can and should improve WHEEL metadata. Perhaps the NumPy/SciPy readers would care to comment, as we're talking beyond the realms of Python version compatibility. If you have other reasons for your -1, I'd like to hear them.
need to use the feature. (Would you be willing to add some sort of "never extract C extensions regardless of what the metadata might say" option to wheel mount? It's not critical, as mounting random wheels without knowing how they work is clearly bad, but it does add a level of assurance that might be helpful,)
Currently, extraction only happens if there is metadata in the wheel to indicate that extraction should occur, and it's up to the wheel builder to put it there. It doesn't make sense in general to prohibit just the C extensions but allow importing pure-Python modules in a wheel: the pure-Python bits mightn't work properly if the C extensions aren't available. So it seems safer in general to have all-or-nothing, or else have an additional flag in the metadata which indicates that the C extensions have to be extracted for the wheel to be useful. End user control of mounting should be at the discretion of the developer of the software which invokes the mount method. Regards, Vinay Sajip
On 30 January 2014 11:57, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:
ISTM the reason why it can't go in there just yet is also the reason why one might have some reservations about the feature: binary compatibility
Sorry, I didn't comment on this point. To me, binary compatibility is *not* the issue. So my other email ignored it. It is a reasonable point, but not one that bothers me, oddly enough :-) Paul
On Thu, Jan 30, 2014 at 3:09 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 30 January 2014 05:33, Evgeny Sazhin <eugene@sazhin.us> wrote:
Eh, I think both 1 and 3 are things that are possibly reasonable to happen and they are both things that I've contemplated as things to bring forward in using xz as an alternative compression format. Even if #1 would need a major revision of Wheel to happen adding official "support" for zip import means that the change would have to be weighed against also breaking that backwards compatibility.
Eh, please don't break it!! Improve! Keep the ability to add wheels to python path. I don't care how, zip import, non-zip import, whatever - allow for jar-like behavior - people will thank you if it will work properly !;) I know, i'm probably alone in the desert...l;)
This is the biggest concern I see with "promoting" wheels being directly importable via zipimport (I say "promoting" and not "specifying" deliberately, but I don't want to get back into the process debate).
People like Evgeny (no offense intended here, but your post is a good example to use) coming from a Java background will see importable wheels and *think* that they are similar in purpose to jars.
No offense taken - i'm happy to be a use case to add some perspective;))
This is not, and never will be, true. Python is not Java, deployment by putting jar-like files on sys.path is not an advisable workflow for Python - no matter what parallels with Java might make it look attractive.
For pure python wheels - you're right - very attractive! And while i hear you guys saying it is not advisable to use wheels in sys.path i'm still far from seeing good reasoning. Please, correct me, but so far i saw 2: 1. It is impossible to load C extensions 2. Error handling is bad None of these reasons seems to be good enough to lead to a conclusion that wheel used as jar-like thing is not needed. Here is how i see it (again probably to my lack of knowledge): 1. Yes there is a group of "elite" interesting projects that are using C extensions together with python code. What i don't understand is why are wheels supposed to be tailored to the needs of the hybrid projects instead of making pure python first priority? There is big crowd using C extensions, but how much bigger is the crowd who doesn't? Why law abiding pure python developer should jump through the hoops? 2. IF we are talking about error handling it seems that people generally do not consider jar-like behavior to be bad for pure python modules - It's just doesn't work very well (yet?). Is there really a problem with error handling for pure python? What if the error handling is improved? Is it still going to be bad? Why?
Pointing out the implication that wheels can be put on sys.path in specialised circumstances is entirely reasonable when talking to a Python audience, but it triggers entirely the wrong associations when a person with a Java background sees the statement.
Evgeny - for your purposes, you'd be better advised to see wheels as similar in principle to Windows MSI installers (they aren't, but it will avoid triggering incorrect analogies).
That is an awesome analogy, thank you!
I don't have good references or pointers to good Python development or deployment practices, but you may want to ask around on python-list.
And that is my biggest concern (deployment). I believe these questions should be answered by wheels or python packaging specification. Not by third party tooling. The language should be able to work with artifacts/packages produced for the language by the language. Third party tools may allow for automation, continuous integration complicated deployments and whatnot. But the base must be there. Thanks, Eugene
(Be careful, you may hear suggestions that you use eggs because they are similar to jars - while that is true, the way Python works means that the experience with eggs is far from being as natural as the jar experience you're used to).
Paul
On Jan 30, 2014, at 11:33 AM, Evgeny Sazhin <eugene@sazhin.us> wrote:
On Thu, Jan 30, 2014 at 3:09 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 30 January 2014 05:33, Evgeny Sazhin <eugene@sazhin.us> wrote:
Eh, I think both 1 and 3 are things that are possibly reasonable to happen and they are both things that I've contemplated as things to bring forward in using xz as an alternative compression format. Even if #1 would need a major revision of Wheel to happen adding official "support" for zip import means that the change would have to be weighed against also breaking that backwards compatibility.
Eh, please don't break it!! Improve! Keep the ability to add wheels to python path. I don't care how, zip import, non-zip import, whatever - allow for jar-like behavior - people will thank you if it will work properly !;) I know, i'm probably alone in the desert...l;)
This is the biggest concern I see with "promoting" wheels being directly importable via zipimport (I say "promoting" and not "specifying" deliberately, but I don't want to get back into the process debate).
People like Evgeny (no offense intended here, but your post is a good example to use) coming from a Java background will see importable wheels and *think* that they are similar in purpose to jars.
No offense taken - i'm happy to be a use case to add some perspective;))
This is not, and never will be, true. Python is not Java, deployment by putting jar-like files on sys.path is not an advisable workflow for Python - no matter what parallels with Java might make it look attractive.
For pure python wheels - you're right - very attractive! And while i hear you guys saying it is not advisable to use wheels in sys.path i'm still far from seeing good reasoning. Please, correct me, but so far i saw 2:
1. It is impossible to load C extensions 2. Error handling is bad
None of these reasons seems to be good enough to lead to a conclusion that wheel used as jar-like thing is not needed. Here is how i see it (again probably to my lack of knowledge):
1. Yes there is a group of "elite" interesting projects that are using C extensions together with python code. What i don't understand is why are wheels supposed to be tailored to the needs of the hybrid projects instead of making pure python first priority? There is big crowd using C extensions, but how much bigger is the crowd who doesn't? Why law abiding pure python developer should jump through the hoops?
Because C extensions are not “elite”, they are pretty common. It’s somewhat rare in my experience to see a non trivial project that doesn’t have a C extension somewhere in it.
2. IF we are talking about error handling it seems that people generally do not consider jar-like behavior to be bad for pure python modules - It's just doesn't work very well (yet?). Is there really a problem with error handling for pure python? What if the error handling is improved? Is it still going to be bad? Why?
i consider the fact that they’ll only work in a subset of situations bad, because it’s non obvious without digging through the code if any particular project will support working from a place other than a traditional install. This isn’t a new feature, Eggs have had this for a long time and when confronted with bugs caused by zipped Eggs the common solution is simply adding zip_safe=False to their project which disables zipped eggs for that project. The name for the kind of feature this is, is an “attractive nuisance”. They look like something that might work, especially to people coming to Python from other languages such as Java, but if actually use the feature in a non trivial fashion you’re going to end up with more problems then you’ve solved when random projects simply don’t work and the likelihood that their authors have no desire to support running through a zipped file even if they *could* make it possible.
Pointing out the implication that wheels can be put on sys.path in specialised circumstances is entirely reasonable when talking to a Python audience, but it triggers entirely the wrong associations when a person with a Java background sees the statement.
Evgeny - for your purposes, you'd be better advised to see wheels as similar in principle to Windows MSI installers (they aren't, but it will avoid triggering incorrect analogies).
That is an awesome analogy, thank you!
I don't have good references or pointers to good Python development or deployment practices, but you may want to ask around on python-list.
And that is my biggest concern (deployment). I believe these questions should be answered by wheels or python packaging specification. Not by third party tooling. The language should be able to work with artifacts/packages produced for the language by the language. Third party tools may allow for automation, continuous integration complicated deployments and whatnot. But the base must be there.
Thanks, Eugene
(Be careful, you may hear suggestions that you use eggs because they are similar to jars - while that is true, the way Python works means that the experience with eggs is far from being as natural as the jar experience you're used to).
Paul
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Jan 30, 2014, at 11:59 AM, Donald Stufft <donald@stufft.io> wrote:
Because C extensions are not “elite”, they are pretty common. It’s somewhat rare in my experience to see a non trivial project that doesn’t have a C extension somewhere in it.
To be clear, I mean project in the application sense here not in the individual thing that you’d upload to PyPI sense. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Thu, Jan 30, 2014 at 11:59 AM, Donald Stufft <donald@stufft.io> wrote:
On Jan 30, 2014, at 11:33 AM, Evgeny Sazhin <eugene@sazhin.us> wrote:
On Thu, Jan 30, 2014 at 3:09 AM, Paul Moore <p.f.moore@gmail.com> wrote:
On 30 January 2014 05:33, Evgeny Sazhin <eugene@sazhin.us> wrote:
Eh, I think both 1 and 3 are things that are possibly reasonable to happen and they are both things that I've contemplated as things to bring forward in using xz as an alternative compression format. Even if #1 would need a major revision of Wheel to happen adding official "support" for zip import means that the change would have to be weighed against also breaking that backwards compatibility.
Eh, please don't break it!! Improve! Keep the ability to add wheels to python path. I don't care how, zip import, non-zip import, whatever - allow for jar-like behavior - people will thank you if it will work properly !;) I know, i'm probably alone in the desert...l;)
This is the biggest concern I see with "promoting" wheels being directly importable via zipimport (I say "promoting" and not "specifying" deliberately, but I don't want to get back into the process debate).
People like Evgeny (no offense intended here, but your post is a good example to use) coming from a Java background will see importable wheels and *think* that they are similar in purpose to jars.
No offense taken - i'm happy to be a use case to add some perspective;))
This is not, and never will be, true. Python is not Java, deployment by putting jar-like files on sys.path is not an advisable workflow for Python - no matter what parallels with Java might make it look attractive.
For pure python wheels - you're right - very attractive! And while i hear you guys saying it is not advisable to use wheels in sys.path i'm still far from seeing good reasoning. Please, correct me, but so far i saw 2:
1. It is impossible to load C extensions 2. Error handling is bad
None of these reasons seems to be good enough to lead to a conclusion that wheel used as jar-like thing is not needed. Here is how i see it (again probably to my lack of knowledge):
1. Yes there is a group of "elite" interesting projects that are using C extensions together with python code. What i don't understand is why are wheels supposed to be tailored to the needs of the hybrid projects instead of making pure python first priority? There is big crowd using C extensions, but how much bigger is the crowd who doesn't? Why law abiding pure python developer should jump through the hoops?
Because C extensions are not "elite", they are pretty common. It's somewhat rare in my experience to see a non trivial project that doesn't have a C extension somewhere in it.
Isn't this a self contradictory statement? C extensions are quite common for non-trivial project. non-trivial is not common. Doesn't it mean that C extensions are rarely used for common trivial stuff? You also implying that there were non-trivial projects that didn't need C extensions. Or are you trying to make an argument that the overall majority of python projects have C extensions? Is python such a bad language it cannot deal with simple stuff without C being involved?
2. IF we are talking about error handling it seems that people generally do not consider jar-like behavior to be bad for pure python modules - It's just doesn't work very well (yet?). Is there really a problem with error handling for pure python? What if the error handling is improved? Is it still going to be bad? Why?
i consider the fact that they'll only work in a subset of situations bad, because it's non obvious without digging through the code if any particular project will support working from a place other than a traditional install.
The necessity to dig through the code to understand if the project is runnable from zip is a responsibility of project provider - i.e. poor documentation. If the project is designed to be run that way the doc should say so.
This isn't a new feature, Eggs have had this for a long time and when confronted with bugs caused by zipped Eggs the common solution is simply adding zip_safe=False to their project which disables zipped eggs for that project.
The name for the kind of feature this is, is an "attractive nuisance". They look like something that might work, especially to people coming to Python from other languages such as Java, but if actually use the feature in a non trivial fashion you're going to end up with more problems then you've solved when random projects simply don't work and the likelihood that their authors have no desire to support running through a zipped file even if they *could* make it possible.
OK, you're saying " if actually use the feature in a non trivial fashion" But was is non-trivial in putting pure python wheel in sys.path? Non-trivial would probably mean trying to load c extension from it, but that's obviously not supported, so i wouldn't even try. Thanks, Eugene
On Thu, Jan 30, 2014 at 12:46 PM, Evgeny Sazhin <eugene@sazhin.us> wrote:
On Thu, Jan 30, 2014 at 11:59 AM, Donald Stufft <donald@stufft.io> wrote:
On Jan 30, 2014, at 11:33 AM, Evgeny Sazhin <eugene@sazhin.us> wrote:
On Thu, Jan 30, 2014 at 3:09 AM, Paul Moore <p.f.moore@gmail.com>
wrote:
On 30 January 2014 05:33, Evgeny Sazhin <eugene@sazhin.us> wrote:
Eh, I think both 1 and 3 are things that are possibly reasonable to happen and they are both things that I've contemplated as things to bring forward in using xz as an alternative compression format. Even if #1 would need a major revision of Wheel to happen adding official "support" for zip import means that the change would have to be weighed against also breaking that backwards compatibility.
Eh, please don't break it!! Improve! Keep the ability to add wheels to python path. I don't care how, zip import, non-zip import, whatever - allow for jar-like behavior - people will thank you if it will work properly !;) I know, i'm probably alone in the desert...l;)
This is the biggest concern I see with "promoting" wheels being directly importable via zipimport (I say "promoting" and not "specifying" deliberately, but I don't want to get back into the process debate).
People like Evgeny (no offense intended here, but your post is a good example to use) coming from a Java background will see importable wheels and *think* that they are similar in purpose to jars.
No offense taken - i'm happy to be a use case to add some perspective;))
This is not, and never will be, true. Python is not Java, deployment by putting jar-like files on sys.path is not an advisable workflow for Python - no matter what parallels with Java might make it look attractive.
For pure python wheels - you're right - very attractive! And while i hear you guys saying it is not advisable to use wheels in sys.path i'm still far from seeing good reasoning. Please, correct me, but so far i saw 2:
1. It is impossible to load C extensions 2. Error handling is bad
None of these reasons seems to be good enough to lead to a conclusion that wheel used as jar-like thing is not needed. Here is how i see it (again probably to my lack of knowledge):
1. Yes there is a group of "elite" interesting projects that are using C extensions together with python code. What i don't understand is why are wheels supposed to be tailored to the needs of the hybrid projects instead of making pure python first priority? There is big crowd using C extensions, but how much bigger is the crowd who doesn't? Why law abiding pure python developer should jump through the hoops?
Because C extensions are not "elite", they are pretty common. It's somewhat rare in my experience to see a non trivial project that doesn't have a C extension somewhere in it.
Isn't this a self contradictory statement? C extensions are quite common for non-trivial project. non-trivial is not common. Doesn't it mean that C extensions are rarely used for common trivial stuff? You also implying that there were non-trivial projects that didn't need C extensions. Or are you trying to make an argument that the overall majority of python projects have C extensions?
He's trying to argue that enough projects use C extensions that to have special support for projects that don't use them is supporting a special case. The Zen of Python says "Special cases aren't special enough to break the rules" and explicitly having wheels support direct execution only when they contain pure Python is supporting a special case. Obviously you can argue over at what point something stops being a "special case", but considering venvs isolate code well and are the officially supported way to isolate code then that makes stuff that goes passed what venv can do a special case.
Is python such a bad language it cannot deal with simple stuff without C being involved?
The use of a C extension does not necessarily mean a shortcoming of Python. People can write performance-critical code in C so as to get that last bit of speed in heavily executed code in a tight loop, etc. It has nothing to do with Python not being able to handle something and more that C code can simply be faster when coded correctly; it's a feature and not a band-aid. -Brett
2. IF we are talking about error handling it seems that people generally do not consider jar-like behavior to be bad for pure python modules - It's just doesn't work very well (yet?). Is there really a problem with error handling for pure python? What if the error handling is improved? Is it still going to be bad? Why?
i consider the fact that they'll only work in a subset of situations bad, because it's non obvious without digging through the code if any particular project will support working from a place other than a traditional install.
The necessity to dig through the code to understand if the project is runnable from zip is a responsibility of project provider - i.e. poor documentation. If the project is designed to be run that way the doc should say so.
This isn't a new feature, Eggs have had this for a long time and when confronted with bugs caused by zipped Eggs the common solution is simply adding zip_safe=False to their project which disables zipped eggs for that project.
The name for the kind of feature this is, is an "attractive nuisance". They look like something that might work, especially to people coming to Python from other languages such as Java, but if actually use the feature in a non trivial fashion you're going to end up with more problems then you've solved when random projects simply don't work and the likelihood that their authors have no desire to support running through a zipped file even if they *could* make it possible.
OK, you're saying " if actually use the feature in a non trivial fashion" But was is non-trivial in putting pure python wheel in sys.path? Non-trivial would probably mean trying to load c extension from it, but that's obviously not supported, so i wouldn't even try.
Thanks, Eugene _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Thu, Jan 30, 2014 at 2:52 PM, Brett Cannon <brett@python.org> wrote:
still far from seeing good reasoning. Please, correct me, but so far i saw 2:
1. It is impossible to load C extensions 2. Error handling is bad
None of these reasons seems to be good enough to lead to a conclusion that wheel used as jar-like thing is not needed. Here is how i see it (again probably to my lack of knowledge):
1. Yes there is a group of "elite" interesting projects that are using C extensions together with python code. What i don't understand is why are wheels supposed to be tailored to the needs of the hybrid projects instead of making pure python first priority? There is big crowd using C extensions, but how much bigger is the crowd who doesn't? Why law abiding pure python developer should jump through the hoops?
Because C extensions are not "elite", they are pretty common. It's somewhat rare in my experience to see a non trivial project that doesn't have a C extension somewhere in it.
Isn't this a self contradictory statement? C extensions are quite common for non-trivial project. non-trivial is not common. Doesn't it mean that C extensions are rarely used for common trivial stuff? You also implying that there were non-trivial projects that didn't need C extensions. Or are you trying to make an argument that the overall majority of python projects have C extensions?
He's trying to argue that enough projects use C extensions that to have special support for projects that don't use them is supporting a special case. The Zen of Python says "Special cases aren't special enough to break the rules" and explicitly having wheels support direct execution only when they contain pure Python is supporting a special case. Obviously you can argue over at what point something stops being a "special case", but considering venvs isolate code well and are the officially supported way to isolate code then that makes stuff that goes passed what venv can do a special case.
yes, you got it absolutely right - I'm coming from the point of view that the C extension is a special case. It's even called extension! And pure python is never special case, therefore should take priority.
Is python such a bad language it cannot deal with simple stuff without C being involved?
The use of a C extension does not necessarily mean a shortcoming of Python. People can write performance-critical code in C so as to get that last bit of speed in heavily executed code in a tight loop, etc. It has nothing to do with Python not being able to handle something and more that C code can simply be faster when coded correctly; it's a feature and not a band-aid.
-Brett
I understand that, i was sarcastic;) But thank you! Thanks, Eugene
On 30 January 2014 16:33, Evgeny Sazhin <eugene@sazhin.us> wrote:
I don't have good references or pointers to good Python development or deployment practices, but you may want to ask around on python-list.
And that is my biggest concern (deployment). I believe these questions should be answered by wheels or python packaging specification. Not by third party tooling. The language should be able to work with artifacts/packages produced for the language by the language. Third party tools may allow for automation, continuous integration complicated deployments and whatnot. But the base must be there.
There's 2 distinct concepts in the Python world, that you should be aware of. Application deployment and library deployment. Library deployment is simple. Build your library and upload it as a source distribution (sdist) and one or more wheels to PyPI (if it's a public project) or a local index (if it's a private project). People or other projects will use the library however they deem appropriate, typically by doing "pip install projectname" to make the code accessible. That unpacks the wheel into a local site-packages, but you have no need to care about that. Application deployment is not as straightforward - there are a number of common approaches and at least as many opinions on what you should do. You may bundle Python and your code and its dependencies together into a standalone executable, using a tool like cx_Freeze. or you may use something like buildout (I can't give details as I've never used this). You can also, in some cases, package your application main code in the same way as a library, and use "entry points" to define the executables - that will then be installed by the end user into an existing Python installation. As an application developer, you typically wouldn't care about how the end user deployed your code for execution, unless you were bundling via something like cx_Freeze. You might choose to bundle your application code *without* a Python interpreter. That's basically what executable zipfiles are about, and it's probably the closest equivalent to Java jars. But that's not a particularly well-travelled route in the Python community (yet) - there are projects like pyzzer that automate certain aspects of that process, but in general, you have to "roll your own" solution here. Hopefully that makes some sense - I'm getting pretty burned out with the various threads going on at the moment, so I'm probably glossing over a lot. Cheers, Paul
On Thu, Jan 30, 2014 at 12:56 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 30 January 2014 16:33, Evgeny Sazhin <eugene@sazhin.us> wrote:
I don't have good references or pointers to good Python development or deployment practices, but you may want to ask around on python-list.
And that is my biggest concern (deployment). I believe these questions should be answered by wheels or python packaging specification. Not by third party tooling. The language should be able to work with artifacts/packages produced for the language by the language. Third party tools may allow for automation, continuous integration complicated deployments and whatnot. But the base must be there.
There's 2 distinct concepts in the Python world, that you should be aware of. Application deployment and library deployment.
Library deployment is simple. Build your library and upload it as a source distribution (sdist) and one or more wheels to PyPI (if it's a public project) or a local index (if it's a private project). People or other projects will use the library however they deem appropriate, typically by doing "pip install projectname" to make the code accessible. That unpacks the wheel into a local site-packages, but you have no need to care about that.
Application deployment is not as straightforward - there are a number of common approaches and at least as many opinions on what you should do. You may bundle Python and your code and its dependencies together into a standalone executable, using a tool like cx_Freeze. or you may use something like buildout (I can't give details as I've never used this). You can also, in some cases, package your application main code in the same way as a library, and use "entry points" to define the executables - that will then be installed by the end user into an existing Python installation.
As an application developer, you typically wouldn't care about how the end user deployed your code for execution, unless you were bundling via something like cx_Freeze. You might choose to bundle your application code *without* a Python interpreter. That's basically what executable zipfiles are about, and it's probably the closest equivalent to Java jars. But that's not a particularly well-travelled route in the Python community (yet) - there are projects like pyzzer that automate certain aspects of that process, but in general, you have to "roll your own" solution here.
Hopefully that makes some sense - I'm getting pretty burned out with the various threads going on at the moment, so I'm probably glossing over a lot.
Cheers, Paul
Thanks Paul - you are very helpful! this is good distinction and i think i'm coming to it as well. I just suggested one approach that might be feasible in my case in another thread;) Everybody - I'm really grateful for the time you are willing to spend in helping me to understand the state of the things. I hope this discussion might be helpful for both sides and may be people like me can be referenced to it in future;) PS That doesn't mean i will stop arguing with you;) Thanks, Eugene
One thing that might be useful would be to develop the "unpacked wheel" which is currently undefined but would be deliberately identical to a site-packages with just one wheel extracted into it. You wouldn't have to argue or worry about the zip issue. I like the way npm puts everything into a directory ~/.npm/packagename/0.4.2/ ... for example, gem is similar. If you really wanted to go to town you could figure out how to do virtualenvs with hardlinks or reflinks instead of copies (conda can). But as has been repeated you won't find robust support for this in the existing code. On Thu, Jan 30, 2014 at 1:19 PM, Evgeny Sazhin <eugene@sazhin.us> wrote:
On Thu, Jan 30, 2014 at 12:56 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 30 January 2014 16:33, Evgeny Sazhin <eugene@sazhin.us> wrote:
I don't have good references or pointers to good Python development or deployment practices, but you may want to ask around on python-list.
And that is my biggest concern (deployment). I believe these questions should be answered by wheels or python packaging specification. Not by third party tooling. The language should be able to work with artifacts/packages produced for the language by the language. Third party tools may allow for automation, continuous integration complicated deployments and whatnot. But the base must be there.
There's 2 distinct concepts in the Python world, that you should be aware of. Application deployment and library deployment.
Library deployment is simple. Build your library and upload it as a source distribution (sdist) and one or more wheels to PyPI (if it's a public project) or a local index (if it's a private project). People or other projects will use the library however they deem appropriate, typically by doing "pip install projectname" to make the code accessible. That unpacks the wheel into a local site-packages, but you have no need to care about that.
Application deployment is not as straightforward - there are a number of common approaches and at least as many opinions on what you should do. You may bundle Python and your code and its dependencies together into a standalone executable, using a tool like cx_Freeze. or you may use something like buildout (I can't give details as I've never used this). You can also, in some cases, package your application main code in the same way as a library, and use "entry points" to define the executables - that will then be installed by the end user into an existing Python installation.
As an application developer, you typically wouldn't care about how the end user deployed your code for execution, unless you were bundling via something like cx_Freeze. You might choose to bundle your application code *without* a Python interpreter. That's basically what executable zipfiles are about, and it's probably the closest equivalent to Java jars. But that's not a particularly well-travelled route in the Python community (yet) - there are projects like pyzzer that automate certain aspects of that process, but in general, you have to "roll your own" solution here.
Hopefully that makes some sense - I'm getting pretty burned out with the various threads going on at the moment, so I'm probably glossing over a lot.
Cheers, Paul
Thanks Paul - you are very helpful! this is good distinction and i think i'm coming to it as well. I just suggested one approach that might be feasible in my case in another thread;)
Everybody - I'm really grateful for the time you are willing to spend in helping me to understand the state of the things. I hope this discussion might be helpful for both sides and may be people like me can be referenced to it in future;)
PS That doesn't mean i will stop arguing with you;)
Thanks, Eugene _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Thu, Jan 30, 2014 at 1:24 PM, Daniel Holth <dholth@gmail.com> wrote:
One thing that might be useful would be to develop the "unpacked wheel" which is currently undefined but would be deliberately identical to a site-packages with just one wheel extracted into it. You wouldn't have to argue or worry about the zip issue.
I like the way npm puts everything into a directory ~/.npm/packagename/0.4.2/ ... for example, gem is similar. If you really wanted to go to town you could figure out how to do virtualenvs with hardlinks or reflinks instead of copies (conda can).
But as has been repeated you won't find robust support for this in the existing code.
I just tested it - it works! Yeah! I put an unzipped wheel of my API project into a local wheelhouse - pip was able to install from it into my virt env for the client project. Then i created a wheel with __main__.py for the client project and put an unzipped version in wheelhouse. After that i issued: PYTHONPATH=~/wheelhouse/projectAPI.whl; python ~/wheelhouse/projectClient.whl It worked! Is this workflow OK to rely on in future? Thanks, Eugene
On Thu, Jan 30, 2014 at 2:15 PM, Evgeny Sazhin <eugene@sazhin.us> wrote:
On Thu, Jan 30, 2014 at 1:24 PM, Daniel Holth <dholth@gmail.com> wrote:
One thing that might be useful would be to develop the "unpacked wheel" which is currently undefined but would be deliberately identical to a site-packages with just one wheel extracted into it. You wouldn't have to argue or worry about the zip issue.
I like the way npm puts everything into a directory ~/.npm/packagename/0.4.2/ ... for example, gem is similar. If you really wanted to go to town you could figure out how to do virtualenvs with hardlinks or reflinks instead of copies (conda can).
But as has been repeated you won't find robust support for this in the existing code.
I just tested it - it works! Yeah!
I put an unzipped wheel of my API project into a local wheelhouse - pip was able to install from it into my virt env for the client project. Then i created a wheel with __main__.py for the client project and put an unzipped version in wheelhouse.
After that i issued: PYTHONPATH=~/wheelhouse/projectAPI.whl; python ~/wheelhouse/projectClient.whl
It worked!
Is this workflow OK to rely on in future?
Thanks, Eugene
The only request is that you don't name the directory ".whl" because that is a little bit confusing. "Directories with only one package installed into them" will continue to work.
On Thu, Jan 30, 2014 at 2:32 PM, Daniel Holth <dholth@gmail.com> wrote:
On Thu, Jan 30, 2014 at 2:15 PM, Evgeny Sazhin <eugene@sazhin.us> wrote:
On Thu, Jan 30, 2014 at 1:24 PM, Daniel Holth <dholth@gmail.com> wrote:
One thing that might be useful would be to develop the "unpacked wheel" which is currently undefined but would be deliberately identical to a site-packages with just one wheel extracted into it. You wouldn't have to argue or worry about the zip issue.
I like the way npm puts everything into a directory ~/.npm/packagename/0.4.2/ ... for example, gem is similar. If you really wanted to go to town you could figure out how to do virtualenvs with hardlinks or reflinks instead of copies (conda can).
But as has been repeated you won't find robust support for this in the existing code.
I just tested it - it works! Yeah!
I put an unzipped wheel of my API project into a local wheelhouse - pip was able to install from it into my virt env for the client project. Then i created a wheel with __main__.py for the client project and put an unzipped version in wheelhouse.
After that i issued: PYTHONPATH=~/wheelhouse/projectAPI.whl; python ~/wheelhouse/projectClient.whl
It worked!
Is this workflow OK to rely on in future?
Thanks, Eugene
The only request is that you don't name the directory ".whl" because that is a little bit confusing.
"Directories with only one package installed into them" will continue to work.
pip can't find the distribution then - so it must have the name totally matching wheel spec, i guess.
Wow, it really can install from a directory-named-like-a-wheel with the --no-index flag passed. I'm sad to say that is totally unintentional, but bizarrely interesting. On Thu, Jan 30, 2014 at 2:52 PM, Evgeny Sazhin <eugene@sazhin.us> wrote:
On Thu, Jan 30, 2014 at 2:32 PM, Daniel Holth <dholth@gmail.com> wrote:
On Thu, Jan 30, 2014 at 2:15 PM, Evgeny Sazhin <eugene@sazhin.us> wrote:
On Thu, Jan 30, 2014 at 1:24 PM, Daniel Holth <dholth@gmail.com> wrote:
One thing that might be useful would be to develop the "unpacked wheel" which is currently undefined but would be deliberately identical to a site-packages with just one wheel extracted into it. You wouldn't have to argue or worry about the zip issue.
I like the way npm puts everything into a directory ~/.npm/packagename/0.4.2/ ... for example, gem is similar. If you really wanted to go to town you could figure out how to do virtualenvs with hardlinks or reflinks instead of copies (conda can).
But as has been repeated you won't find robust support for this in the existing code.
I just tested it - it works! Yeah!
I put an unzipped wheel of my API project into a local wheelhouse - pip was able to install from it into my virt env for the client project. Then i created a wheel with __main__.py for the client project and put an unzipped version in wheelhouse.
After that i issued: PYTHONPATH=~/wheelhouse/projectAPI.whl; python ~/wheelhouse/projectClient.whl
It worked!
Is this workflow OK to rely on in future?
Thanks, Eugene
The only request is that you don't name the directory ".whl" because that is a little bit confusing.
"Directories with only one package installed into them" will continue to work.
pip can't find the distribution then - so it must have the name totally matching wheel spec, i guess.
On Thu, Jan 30, 2014 at 3:04 PM, Daniel Holth <dholth@gmail.com> wrote:
Wow, it really can install from a directory-named-like-a-wheel with the --no-index flag passed. I'm sad to say that is totally unintentional, but bizarrely interesting.
Can you guys promiss at least not to break this one?!;) Thanks, Eugene
-------------------------------------------- On Thu, 30/1/14, Daniel Holth <dholth@gmail.com> wrote:
Wow, it really can install from a directory-named-like-a-wheel with the --no-index flag passed. I'm sad to say that is totally unintentional, but bizarrely interesting.
What, *more* unintentional drive-by behaviour? Now, where's that pitchfork? ;-) Regards, Vinay Sajip
participants (6)
-
Brett Cannon -
Daniel Holth -
Donald Stufft -
Evgeny Sazhin -
Paul Moore -
Vinay Sajip