I really want to use eggs, but I *have* to use py2app. Basically, the issues that I see are:
1) modulegraph needs to know about eggs so that it can track down the dependencies of things in the eggs
2) the eggs should either be put into the site-packages.zip file in the bundle in "basket" form, or should be tossed directly into the site-packages directory.
For the first point, I'm curious if modulegraph already knows about zip packages. I didn't see anything in the code that immediately jumped out at me saying "hey! this package is zipped up". What I had seen was that the contents of eggs were just being ignored. If module graph does have some knowledge about zip libraries, that would probably translate well.
As for #2, the ideal would probably be to put everything in basket form in site-packages.zip. I figure that by opening up the eggs and only pulling out the referenced parts, the overall app can be smaller. Any package resources need to get in there, though. For expedience, though, I'm quite find with a little bit larger app that has the .egg files stored directly in site-packages with a .pth file.
Any commentary on the difficulty involved here? (Bob?)
Thanks, Kevin
On Jul 12, 2005, at 4:51 PM, Kevin Dangoor wrote:
I really want to use eggs, but I *have* to use py2app. Basically, the issues that I see are:
- modulegraph needs to know about eggs so that it can track down the
dependencies of things in the eggs
- the eggs should either be put into the site-packages.zip file in
the bundle in "basket" form, or should be tossed directly into the site-packages directory.
For the first point, I'm curious if modulegraph already knows about zip packages. I didn't see anything in the code that immediately jumped out at me saying "hey! this package is zipped up". What I had seen was that the contents of eggs were just being ignored. If module graph does have some knowledge about zip libraries, that would probably translate well.
As for #2, the ideal would probably be to put everything in basket form in site-packages.zip. I figure that by opening up the eggs and only pulling out the referenced parts, the overall app can be smaller. Any package resources need to get in there, though. For expedience, though, I'm quite find with a little bit larger app that has the .egg files stored directly in site-packages with a .pth file.
Any commentary on the difficulty involved here? (Bob?)
The most expedient option would be to specify the eggs and a pth as resources (or data_files) manually.
Second to that, you could make sure all the eggs are unzipped and on sys.path when the setup.py is running (this doesn't solve the resources thing, but modulegraph doesn't do that anyway).
As far as fixing the implementation, I don't know.. I don't really have time to work on open source Python stuff right now. Patches accepted, of course.
-bob
On 7/12/05, Bob Ippolito bob@redivi.com wrote:
The most expedient option would be to specify the eggs and a pth as resources (or data_files) manually.
Second to that, you could make sure all the eggs are unzipped and on sys.path when the setup.py is running (this doesn't solve the resources thing, but modulegraph doesn't do that anyway).
As far as fixing the implementation, I don't know.. I don't really have time to work on open source Python stuff right now. Patches accepted, of course.
That's why I was asking about complexity... I'm considering digging in myself, but I haven't touched py2app's code before.
I was all in favor of the most expedient option that you mention above. However, the problem that I ran into was that standard library modules that were needed by the eggs weren't getting picked up (since the eggs were just "data files"). If there is an easy way to run modulegraph on the eggs, then maybe a short-term solution is:
- add an option to specify required eggs (using the setuptools syntax, ie "FooBar >= 27.2") - grab these eggs and drop them into the generated site-packages dir with a pth file - scan them for any modules they need and add those to the overall dependency graph (this is where I'm fuzziest on the difficulty level)
So, really... the main problem to solve is finding out what is required by the modules in a given egg.
Kevin
On Jul 12, 2005, at 5:18 PM, Kevin Dangoor wrote:
On 7/12/05, Bob Ippolito bob@redivi.com wrote:
The most expedient option would be to specify the eggs and a pth as resources (or data_files) manually.
Second to that, you could make sure all the eggs are unzipped and on sys.path when the setup.py is running (this doesn't solve the resources thing, but modulegraph doesn't do that anyway).
As far as fixing the implementation, I don't know.. I don't really have time to work on open source Python stuff right now. Patches accepted, of course.
That's why I was asking about complexity... I'm considering digging in myself, but I haven't touched py2app's code before.
I really don't know, I haven't looked at the code recently, especially the modulegraph stuff.
I was all in favor of the most expedient option that you mention above. However, the problem that I ran into was that standard library modules that were needed by the eggs weren't getting picked up (since the eggs were just "data files"). If there is an easy way to run modulegraph on the eggs, then maybe a short-term solution is:
Well, just do the second suggestion then. "Install" the eggs somewhere, unzipped, and make sure it's on sys.path during setup.py. Maybe you could crank the ez_setup wheels (or whatever module is appropriate, I haven't looked at eggs yet) at the top of your setup.py and automate this.
-bob
At 05:29 PM 7/12/2005 -1000, Bob Ippolito wrote:
Well, just do the second suggestion then. "Install" the eggs somewhere, unzipped, and make sure it's on sys.path during setup.py. Maybe you could crank the ez_setup wheels (or whatever module is appropriate, I haven't looked at eggs yet) at the top of your setup.py and automate this.
Well, if it just has to be on sys.path during setup.py, then it should be working now, shouldn't it? I mean, if he installed them to Python's site-packages. In the alternative, he could create a wrapper command that calls require("project==version") to find all the eggs and put them on sys.path.
So, I guess my suggestion to Kevin is to try installing to a directory on sys.path, using the new --always-unzip option to force the packages to be unzipped, and then try hacking py2app to do a require() before looking for dependencies (consulting the setuptools "test" command for an example).
I suspect this means that the packages will get merged into one another with no distinction at runtime, but whatever works, works.
One interesting thing that eggs do, however, in the problem of module-finding is that they potentially reduce the question to one of just identifying what portions of the standard library each egg uses, which in some ways is a significantly easier problem because import tricks (e.g. dynamic imports, exec's, etc.) involving the standard library occur far less often. It might be worth adding something modulegraph-ish to setuptools to support this, although it's not at the top of my list right now.
On Jul 12, 2005, at 6:28 PM, Phillip J. Eby wrote:
At 05:29 PM 7/12/2005 -1000, Bob Ippolito wrote:
Well, just do the second suggestion then. "Install" the eggs somewhere, unzipped, and make sure it's on sys.path during setup.py. Maybe you could crank the ez_setup wheels (or whatever module is appropriate, I haven't looked at eggs yet) at the top of your setup.py and automate this.
Well, if it just has to be on sys.path during setup.py, then it should be working now, shouldn't it? I mean, if he installed them to Python's site-packages. In the alternative, he could create a wrapper command that calls require("project==version") to find all the eggs and put them on sys.path.
Like modulefinder, modulegraph doesn't know anything about fancy PEP 302 stuff yet.
So, I guess my suggestion to Kevin is to try installing to a directory on sys.path, using the new --always-unzip option to force the packages to be unzipped, and then try hacking py2app to do a require() before looking for dependencies (consulting the setuptools "test" command for an example).
I'm not sure you'd really want to hack py2app anyway, just the setup.py before setup() is called.
I suspect this means that the packages will get merged into one another with no distinction at runtime, but whatever works, works.
Yeah, I'm not sure why you would possibly care about this at runtime for an application that's "frozen", anyway.
One interesting thing that eggs do, however, in the problem of module-finding is that they potentially reduce the question to one of just identifying what portions of the standard library each egg uses, which in some ways is a significantly easier problem because import tricks (e.g. dynamic imports, exec's, etc.) involving the standard library occur far less often. It might be worth adding something modulegraph-ish to setuptools to support this, although it's not at the top of my list right now.
Modulegraph should already know about all of the tricks in the stdlib (at least in 2.3, not sure if I updated it for 2.4) including the modules that are imported by the core behind your back (warnings, etc.).. It doesn't contain anything py2app specific, it might as well be used as-is.
http://svn.red-bean.com/bob/py2app/trunk/src/ (modulegraph depends on altgraph, the other stuff you can ignore)
-bob
At 06:42 PM 7/12/2005 -1000, Bob Ippolito wrote:
On Jul 12, 2005, at 6:28 PM, Phillip J. Eby wrote:
At 05:29 PM 7/12/2005 -1000, Bob Ippolito wrote:
Well, just do the second suggestion then. "Install" the eggs somewhere, unzipped, and make sure it's on sys.path during setup.py. Maybe you could crank the ez_setup wheels (or whatever module is appropriate, I haven't looked at eggs yet) at the top of your setup.py and automate this.
Well, if it just has to be on sys.path during setup.py, then it should be working now, shouldn't it? I mean, if he installed them to Python's site-packages. In the alternative, he could create a wrapper command that calls require("project==version") to find all the eggs and put them on sys.path.
Like modulefinder, modulegraph doesn't know anything about fancy PEP 302 stuff yet.
I meant, if he installed with --always-unzip to the system site-packages directory, such that EasyInstall put the needed paths in easy-install.pth.
So, I guess my suggestion to Kevin is to try installing to a directory on sys.path, using the new --always-unzip option to force the packages to be unzipped, and then try hacking py2app to do a require() before looking for dependencies (consulting the setuptools "test" command for an example).
I'm not sure you'd really want to hack py2app anyway, just the setup.py before setup() is called.
Er, no, because then he'd have to change the setup script between building the package and making an app out of it. It needs to be something that runs as a command, *not* something in the setup script.
I suspect this means that the packages will get merged into one another with no distinction at runtime, but whatever works, works.
Yeah, I'm not sure why you would possibly care about this at runtime for an application that's "frozen", anyway.
If the application is composed of plugins, you'd want the plugins' metadata to be available, so there'd be additional work involved to get that put back together.
Modulegraph should already know about all of the tricks in the stdlib (at least in 2.3, not sure if I updated it for 2.4) including the modules that are imported by the core behind your back (warnings, etc.).. It doesn't contain anything py2app specific, it might as well be used as-is.
http://svn.red-bean.com/bob/py2app/trunk/src/ (modulegraph depends on altgraph, the other stuff you can ignore)
Interesting. For py2exe this stuff makes some sense to me, but for py2app I wonder if you really even care about finding stdlib dependencies anyway, if your target Mac OS version has the stdlib already installed. It seems that in that case you could just use something like:
setup.py easy_install --always-unzip --always-copy --install-dir=appdir .
to dump all the unzipped eggs and scripts needed by the current project into 'appdir', and then do whatever Mac OS magic is needed to turn the directory into an application. Oh, and you'd also need to throw in a copy of pkg_resources, I suppose, since it's not part of Python... yet. :)
This approach isn't really an option for py2exe, since there's no version of Windows that bundles Python, but for the many Unix-y operating systems (including Mac OS) that provide Python as a standard feature, it's not a bad choice.
On Jul 12, 2005, at 6:59 PM, Phillip J. Eby wrote:
At 06:42 PM 7/12/2005 -1000, Bob Ippolito wrote:
On Jul 12, 2005, at 6:28 PM, Phillip J. Eby wrote:
At 05:29 PM 7/12/2005 -1000, Bob Ippolito wrote:
Well, just do the second suggestion then. "Install" the eggs somewhere, unzipped, and make sure it's on sys.path during setup.py. Maybe you could crank the ez_setup wheels (or whatever module is appropriate, I haven't looked at eggs yet) at the top of your setup.py and automate this.
Well, if it just has to be on sys.path during setup.py, then it should be working now, shouldn't it? I mean, if he installed them to Python's site-packages. In the alternative, he could create a wrapper command that calls require("project==version") to find all the eggs and put them on sys.path.
Like modulefinder, modulegraph doesn't know anything about fancy PEP 302 stuff yet.
I meant, if he installed with --always-unzip to the system site- packages directory, such that EasyInstall put the needed paths in easy-install.pth.
So, I guess my suggestion to Kevin is to try installing to a directory on sys.path, using the new --always-unzip option to force the packages to be unzipped, and then try hacking py2app to do a require() before looking for dependencies (consulting the setuptools "test" command for an example).
I'm not sure you'd really want to hack py2app anyway, just the setup.py before setup() is called.
Er, no, because then he'd have to change the setup script between building the package and making an app out of it. It needs to be something that runs as a command, *not* something in the setup script.
Right, well, there's easy enough ways around that... subclass away :)
I suspect this means that the packages will get merged into one another with no distinction at runtime, but whatever works, works.
Yeah, I'm not sure why you would possibly care about this at runtime for an application that's "frozen", anyway.
If the application is composed of plugins, you'd want the plugins' metadata to be available, so there'd be additional work involved to get that put back together.
This is only really a concern for people doing egg-centric development (hah). I don't think this is going to be a problem just yet :)
Modulegraph should already know about all of the tricks in the stdlib (at least in 2.3, not sure if I updated it for 2.4) including the modules that are imported by the core behind your back (warnings, etc.).. It doesn't contain anything py2app specific, it might as well be used as-is.
http://svn.red-bean.com/bob/py2app/trunk/src/ (modulegraph depends on altgraph, the other stuff you can ignore)
Interesting. For py2exe this stuff makes some sense to me, but for py2app I wonder if you really even care about finding stdlib dependencies anyway, if your target Mac OS version has the stdlib already installed. It seems that in that case you could just use something like:
setup.py easy_install --always-unzip --always-copy --install- dir=appdir .
to dump all the unzipped eggs and scripts needed by the current project into 'appdir', and then do whatever Mac OS magic is needed to turn the directory into an application. Oh, and you'd also need to throw in a copy of pkg_resources, I suppose, since it's not part of Python... yet. :)
This approach isn't really an option for py2exe, since there's no version of Windows that bundles Python, but for the many Unix-y operating systems (including Mac OS) that provide Python as a standard feature, it's not a bad choice.
You really don't want to depend on the OS installed Python. The biggest reason is because Python 2.3 is going to disappear in 10.5, so your application breaks.
-bob
On 7/12/05, Bob Ippolito bob@redivi.com wrote:
On Jul 12, 2005, at 5:18 PM, Kevin Dangoor wrote:
That's why I was asking about complexity... I'm considering digging in myself, but I haven't touched py2app's code before.
I really don't know, I haven't looked at the code recently, especially the modulegraph stuff.
OK.
Well, just do the second suggestion then. "Install" the eggs somewhere, unzipped, and make sure it's on sys.path during setup.py. Maybe you could crank the ez_setup wheels (or whatever module is appropriate, I haven't looked at eggs yet) at the top of your setup.py and automate this.
I hadn't realized that you haven't looked at eggs yet. I remembered reading on Phillip's blog that you had been involved in the early work and thought you might have continued following it.
I'll take a quick look to see if I can conveniently add the egg's modules to the module graph. Failing that, I'll just work it the way that you and Phillip are suggesting.
Kevin
Either something has changed (for the better!) or I'm losing my mind...
When I first started playing with the eggs (setuptools 0.5a5, I think), py2app pretty much ignored what was in there. This morning, I removed the egg expansion stuff from my setup script and voila! With setuptools 0.5a12, py2app spots the modules in the eggs and handles them just like any other!
So, I just need to figure out what to do about included resources and I'm all set.
Kevin
At 11:15 AM 7/13/2005 -0400, Kevin Dangoor wrote:
Either something has changed (for the better!) or I'm losing my mind...
When I first started playing with the eggs (setuptools 0.5a5, I think), py2app pretty much ignored what was in there. This morning, I removed the egg expansion stuff from my setup script and voila! With setuptools 0.5a12, py2app spots the modules in the eggs and handles them just like any other!
You might clean up any build directories involved, just in case there's residue from when you had the expansion code, because you might be getting a false positive success right now. Just a thought.
On 7/13/05, Phillip J. Eby pje@telecommunity.com wrote:
At 11:15 AM 7/13/2005 -0400, Kevin Dangoor wrote:
Either something has changed (for the better!) or I'm losing my mind...
When I first started playing with the eggs (setuptools 0.5a5, I think), py2app pretty much ignored what was in there. This morning, I removed the egg expansion stuff from my setup script and voila! With setuptools 0.5a12, py2app spots the modules in the eggs and handles them just like any other!
You might clean up any build directories involved, just in case there's residue from when you had the expansion code, because you might be getting a false positive success right now. Just a thought.
Before sending that message this morning, I had poked around in my build directory and didn't see the packages there... apparently, they were there though. I deleted the whole build directory and I'm now getting the missing std lib modules problem that I was before. Oh well.
participants (3)
-
Bob Ippolito -
Kevin Dangoor -
Phillip J. Eby