Hey Robert. I temporariliy moved random back under basic for now. I did this because it's really important to me that rand, randn, fft, and ifft are in the scipy namespace. I don't want you to think I'm unsupportive of the general changes you are trying to see, though. While in principle it would be nice to not do any import magic. There are three problems that I'd like to see solved that I think the import magic is trying to accomplish: 1) making some names visible in the scipy namespace (like fft, rand, etc.) 2) having some mechanism for scipy_core to use the tools in full scipy should it be installed (like svd, for example). 3) having some means to speed up importing. I think that there are two views of how scipy should behave that need to be resolved. The first view is where scipy is at right now in that when the user imports scipy the whole kit-and-kaboodle gets imported at the same time. This view was developed back when it was envisoned that scipy alone would be the MATLAB-like replacement. I think it is more clear now that IPython is better for interactive work and Matplotlib (and Enthought's coming tools) are better for plotting, that something else should provide the "environment for computing" besides the simple command import scipy. The second view is that scipy should just be a simple name-space package and not try to do any magic. Users will primarily use scipy for writing code and will import the needed functions from the right places. Most of what scipy is has been done to support interactive work, so that one does not have to do: import scipy import scipy.integrate import scipy.special in order to get access to the sub-packages. Perhaps this should be re-thought and something like this magic moved to IPython. -Travis
On Fri, 16 Dec 2005, Travis Oliphant wrote:
I temporariliy moved random back under basic for now. I did this because it's really important to me that rand, randn, fft, and ifft are in the scipy namespace. I don't want you to think I'm unsupportive of the general changes you are trying to see, though.
random.py does not need to be under basic to make rand, randn to appear in scipy namespace. When moving random.py to scipy/, the full scipy needs some fixes too, several modules do 'from scipy.basic.random import ..'.
While in principle it would be nice to not do any import magic. There are three problems that I'd like to see solved that I think the import magic is trying to accomplish:
1) making some names visible in the scipy namespace (like fft, rand, etc.)
see a comment at the end of this message..
2) having some mechanism for scipy_core to use the tools in full scipy should it be installed (like svd, for example). 3) having some means to speed up importing.
The speed of importing full scipy without ppimport hooks has been improved a lot when using Python2.3 and up (and faster computers). I myself do not consider the import speed a big issue anymore.
I think that there are two views of how scipy should behave that need to be resolved.
The first view is where scipy is at right now in that when the user imports scipy the whole kit-and-kaboodle gets imported at the same time. This view was developed back when it was envisoned that scipy alone would be the MATLAB-like replacement. I think it is more clear now that IPython is better for interactive work and Matplotlib (and Enthought's coming tools) are better for plotting, that something else should provide the "environment for computing" besides the simple command import scipy.
The second view is that scipy should just be a simple name-space package and not try to do any magic. Users will primarily use scipy for writing code and will import the needed functions from the right places.
Most of what scipy is has been done to support interactive work, so that one does not have to do:
import scipy import scipy.integrate import scipy.special
in order to get access to the sub-packages. Perhaps this should be re-thought and something like this magic moved to IPython.
I just realized that scipy could do similar to Maple when importing packages that provide global symbols. For example, when executing import scipy.fftpack in interactive shell then code like (untested, unoptimized) from scipy.distutils.misc_util import get_frame if get_frame(1).f_locals['__name__']=='__main__': # expose fft to interactive shell exec 'fft = scipy.fftpack.fft' in get_frame(1).f_locals,get_frame(1).f_globals at the end of fftpack/__init__.py would expose fft function to interactive shell (Maple would show also a warning on changing global namespace if I recall correctly). Pearu
Pearu Peterson wrote:
I just realized that scipy could do similar to Maple when importing packages that provide global symbols.
For example, when executing
import scipy.fftpack
in interactive shell then code like (untested, unoptimized)
from scipy.distutils.misc_util import get_frame if get_frame(1).f_locals['__name__']=='__main__': # expose fft to interactive shell exec 'fft = scipy.fftpack.fft' in get_frame(1).f_locals,get_frame(1).f_globals
at the end of fftpack/__init__.py would expose fft function to interactive shell (Maple would show also a warning on changing global namespace if I recall correctly).
-1 on this: for one thing, the caller's interactive namespace is not necessarily one frame above in the stack, so this is fairly brittle. It wouldn't work in ipython, for example, as the user's locals/globals are internally managed dicts and not the frame object's f_locals/f_globals. It would also fail on shells that have more levels of indirection, because __name__ != '__main__' until you actually reach the command-line script that invoked the shell. I also dislike this 'automatic promotion' from a simple import statement, too much implicit behavior for my taste. I think that the import_all() solution, along with Gary's idea, provide a cleaner solution to these issues, which is also (a big plus IMHO) actually robust. Cheers, f
I don't know if this is a stupid idea, but if the vote if for the magic to be moved to ipython, perhaps you could create a subpackage whose sole purpose was provide access to all subpackages in one hit a'la from scipy.interactive import * Gary R. Travis Oliphant wrote:
Hey Robert.
I temporariliy moved random back under basic for now. I did this because it's really important to me that rand, randn, fft, and ifft are in the scipy namespace. I don't want you to think I'm unsupportive of the general changes you are trying to see, though.
While in principle it would be nice to not do any import magic. There are three problems that I'd like to see solved that I think the import magic is trying to accomplish:
1) making some names visible in the scipy namespace (like fft, rand, etc.) 2) having some mechanism for scipy_core to use the tools in full scipy should it be installed (like svd, for example). 3) having some means to speed up importing.
I think that there are two views of how scipy should behave that need to be resolved.
The first view is where scipy is at right now in that when the user imports scipy the whole kit-and-kaboodle gets imported at the same time. This view was developed back when it was envisoned that scipy alone would be the MATLAB-like replacement. I think it is more clear now that IPython is better for interactive work and Matplotlib (and Enthought's coming tools) are better for plotting, that something else should provide the "environment for computing" besides the simple command import scipy.
The second view is that scipy should just be a simple name-space package and not try to do any magic. Users will primarily use scipy for writing code and will import the needed functions from the right places.
Most of what scipy is has been done to support interactive work, so that one does not have to do:
import scipy import scipy.integrate import scipy.special
in order to get access to the sub-packages. Perhaps this should be re-thought and something like this magic moved to IPython.
-Travis
Gary Ruben wrote:
I don't know if this is a stupid idea, but if the vote if for the magic to be moved to ipython, perhaps you could create a subpackage whose sole purpose was provide access to all subpackages in one hit a'la from scipy.interactive import *
I think this is a good idea, but I'd implement it just a little differently. The convenience of scipy.* can be great for code that uses a lot of scipy, so you don't have to type import scipy.this, scipy.that, scipy.that_else,... But I tend to agree with R. Kern that the black magic played by scipy's delayed importer tends to (at least it has in the past, though in fairness it's currently pretty much OK) cause problems. I also think that even though scipy starts pretty fast these days, it would still be a good idea to keep its init time to a minimum, especially for software that is run many times with a fresh process (like unit tests, for example). I'd suggest having in scipy's top level a single method import_all, so that one could write: import scipy scipy.import_all() and from then on use: scipy.this.foo() + scipy.that.bar() This simple idiom could be used in most code that needs to access a lot of scipy, and yet it doesn't pollute the user's global namespace. If you really want a full top-level import, we could have an 'all' module: from scipy.all import * the 'all' module would be the one with the heavy-duty imports. I'd name it 'all' instead of 'interactive' because I think this describes better the fact that it contains 'all of scipy' (though as a matter of suggesting policy by naming, interactive is in fact better :) With this in place, it's a trivial matter for any frequent scipy users to add to their ipython profile the above line (along with matploblib imports and anything else they may want). This means it would work without any changes needed on ipython's side. Note that I'm very happy to make any modifications needed to improve the interactive experience, but I like it even better if we can find solutions that don't require ipython changes (so they also benefit users of other interactive systems, including the plain python '>>>' shell). Cheers, f
I think this is a good idea, but I'd implement it just a little differently. The convenience of scipy.* can be great for code that uses a lot of scipy, so you don't have to type
import scipy.this, scipy.that, scipy.that_else,...
Right this is the reason for the current behavior.
But I tend to agree with R. Kern that the black magic played by scipy's delayed importer tends to (at least it has in the past, though in fairness it's currently pretty much OK) cause problems.
I also think that even though scipy starts pretty fast these days, it would still be a good idea to keep its init time to a minimum, especially for software that is run many times with a fresh process (like unit tests, for example).
I'd suggest having in scipy's top level a single method import_all, so that one could write:
import scipy scipy.import_all()
and from then on use:
scipy.this.foo() + scipy.that.bar()
I like this approach quite a bit. That way coders could choose packages as desired. And interactive use could be made easy as well. What do you think Pearu? You're probably the best one suited to make it happen. -Travis
On Fri, 16 Dec 2005, Travis Oliphant wrote:
But I tend to agree with R. Kern that the black magic played by scipy's delayed importer tends to (at least it has in the past, though in fairness it's currently pretty much OK) cause problems.
I also think that even though scipy starts pretty fast these days, it would still be a good idea to keep its init time to a minimum, especially for software that is run many times with a fresh process (like unit tests, for example).
I agree.
I'd suggest having in scipy's top level a single method import_all, so that one could write:
import scipy scipy.import_all()
and from then on use:
scipy.this.foo() + scipy.that.bar()
I like this approach quite a bit. That way coders could choose packages as desired. And interactive use could be made easy as well.
What do you think Pearu? You're probably the best one suited to make it happen.
Instead of import scipy scipy.import_all() we could have import scipy.import_all or similar such as `import scipy.all` etc. Though having a function import_all() would have an advantage of specifying packages one wishes to see in scipy namespace: import_all('linalg','integrate') would import only scipy.linalg and scipy.integrate packages and their dependencies if there are any. import_all() would import all scipy packages. Hmm, may be scipy.import_all should read as scipy.import and scipy.import() would be scipy.import('all'). However, using Python keyword `import` as a function name would be showstopper here, just import_all does not sound right when it could have optional arguments. Suggestions for better names are welcome. I have another question: what would be in scipy namespace when one types import scipy ? base, test, basic modules. What else? Pearu
Pearu Peterson wrote:
What do you think Pearu? You're probably the best one suited to make it happen.
Instead of
import scipy scipy.import_all()
we could have
import scipy.import_all
or similar such as `import scipy.all` etc.
Though having a function import_all() would have an advantage of specifying packages one wishes to see in scipy namespace: import_all('linalg','integrate') would import only scipy.linalg and scipy.integrate packages and their dependencies if there are any.
import_all() would import all scipy packages. Hmm, may be scipy.import_all should read as scipy.import and scipy.import() would be scipy.import('all'). However, using Python keyword `import` as a function name would be showstopper here, just import_all does not sound right when it could have optional arguments. Suggestions for better names are welcome.
What I don't like about having a module that magically, upon import, rebuilds the scipy namespace, is the silent overloading of the default python semantics for the import statement. It's one more thing to explain: 'this magic module does this and that under the hood just from being imported, ...'. I think that a function approach is clearer and cleaner: its docstring can explicitly specify what it does and doesn't. Thinking of names, I also think it's better to rename it something other than 'import*', so that we distinguish the fact that it's more than an import statement, but rather a namespeace loader. Here's a suggested function header (rough cut): def mod_load(*args): """Load one or more modules into scipy's top-level namespace. Usage: This function is intended to shorten the need to import many of scipy's submodules constantly with statements such as import scipy.linalg, scipy.fft, scipy.etc... Instead, you can say: import scipy scipy.mod_load('linalg','fft',...) or scipy.mod_load('all') to load all of them in one call. If a name which doesn't exist (except for 'all') in scipy's namespace is given, an exception [[WHAT? ImportError, probably?]] is raised. Inputs: - the names (one or more strings) of all the scipy modules one wishes to load into the top-level namespace. If the single argument 'all' is given, then all of scipy's subpackages are imported. Outputs: The function returns a tuple with all the names of the modules which were actually imported. """ And speaking of the 'all' module for interactive use, why can't we just populate the __all__ attribute of scipy's __init__.py without performing any actual imports? With this, a regular from scipy import * will work, as everything listed in __all__ will get imported, but the plain 'import scipy' will still be fast, since __all__ is just a list of strings, it doesn't contain the actual module objects. Granted, with this you can't do import scipy scipy.linalg.foo() but that's precisely why we have scipy.mod_load(). So in summary, unless I'm missing something, with: 1. mod_load as indicated above 2. __all__ properly populated it seems to me we'd have all the desired usage cases covered, no? Cheers, f
Hi, I have implemented hooks for scipy.pkgload(..) following the idea of Fernando's scipy.mod_load(..) functions with few modifications. It's doc string is given below. I have renamed mod_load to pkgload because we have packages. In addition to importing the packages to scipy namespace, scipy.pkgload fills also scipy.__all__ list. Hooks for filling scipy.__doc__ remains to be implemented, it's not difficult though. Currently scipy.pkgload is a bit more verbose than `import scipy` used to be but all the messages are sent to sys.stderr and some of these messages indicate that other parts in scipy needs to be fixed, such as overwriting scipy.test module with ScipyTest instance method etc. Also, `import scipy` still does the full import but after scipy.pkgload is stabilized, importing scipy the following will happen in scipy name space (according to current info.py files): import test from test import ScipyTest import base from base import * import basic from basic import fft, ifft, rand, randn, linalg, fftpack, random I would be interested to know which of these import statements are important to people and which can be removed. For example, what would be the recommended way to access array facilities: from scipy.base import * or from scipy import * ? If the former, then we could remove `from base import *` statement. To import full scipy, one needs to execute import scipy scipy.pkgload() Regards, Pearu Type: instance Base Class: scipy.PackageLoader String Form: <scipy.PackageLoader instance at 0x403a21ac> Namespace: Interactive Constructor Docstring: Manages loading SciPy packages. Callable: Yes Call def: scipy.pkgload(self, *packages, **options) Call docstring: Load one or more packages into scipy's top-level namespace. Usage: This function is intended to shorten the need to import many of scipy's subpackages constantly with statements such as import scipy.linalg, scipy.fft, scipy.etc... Instead, you can say: import scipy scipy.pkgload('linalg','fft',...) or scipy.pkgload() to load all of them in one call. If a name which doesn't exist in scipy's namespace is given, an exception [[WHAT? ImportError, probably?]] is raised. [NotImplemented] Inputs: - the names (one or more strings) of all the scipy modules one wishes to load into the top-level namespace. Optional keyword inputs: - verbose - integer specifying verbosity level [default: 0]. - force - when True, force reloading loaded packages [default: False]. If no input arguments are given, then all of scipy's subpackages are imported. Outputs: The function returns a tuple with all the names of the modules which were actually imported. [NotImplemented] EOM
Pearu Peterson wrote:
I have implemented hooks for scipy.pkgload(..) following the idea of Fernando's scipy.mod_load(..) functions with few modifications. It's doc string is given below.
Well done!
... after scipy.pkgload is stabilized, importing scipy the following will happen in scipy name space (according to current info.py files):
import test from test import ScipyTest import base from base import * import basic from basic import fft, ifft, rand, randn, linalg, fftpack, random
I would be interested to know which of these import statements are important to people and which can be removed.
Am I right in thinking that fft, ifft, rand, and randn are functions, while linalg, fftpack, and random are modules? If so, I'd vote for importing the functions but leaving the modules for an explicit pkgload() command.
For example, what would be the recommended way to access array facilities:
from scipy.base import *
or
from scipy import *
? If the former, then we could remove `from base import *` statement.
I think the latter is friendlier ;) -- Ed
Pearu Peterson wrote:
Hi,
I have implemented hooks for scipy.pkgload(..) following the idea of Fernando's scipy.mod_load(..) functions with few modifications. It's doc string is given below.
Hi Pearu, This is great. I've made a few changes to allow the new system to work with setuptools. Basically, the new patches don't assume that the package __path__ variable contains only a single directory, but rather searches all directories in __path__ for subpackages and modules. This allows scipy_core and full scipy to both exist as .eggs without any other changes. I hope you can commit this patch or something similar before the release this coming week. Cheers! Andrew
On Sat, 31 Dec 2005, Andrew Straw wrote:
This is great. I've made a few changes to allow the new system to work with setuptools. Basically, the new patches don't assume that the package __path__ variable contains only a single directory, but rather searches all directories in __path__ for subpackages and modules. This allows scipy_core and full scipy to both exist as .eggs without any other changes. I hope you can commit this patch or something similar before the release this coming week.
Thanks for the patch. I have applied the patch with modifications. I haven't test it with eggs, so, could give a try. Thanks! Pearu
Pearu Peterson wrote:
On Sat, 31 Dec 2005, Andrew Straw wrote:
This is great. I've made a few changes to allow the new system to work with setuptools. Basically, the new patches don't assume that the package __path__ variable contains only a single directory, but rather searches all directories in __path__ for subpackages and modules. This allows scipy_core and full scipy to both exist as .eggs without any other changes. I hope you can commit this patch or something similar before the release this coming week.
Thanks for the patch. I have applied the patch with modifications. I haven't test it with eggs, so, could give a try.
It works fine. Thanks. Incidentally, this means that eggs with "namespace packages" will work with scipy now. In other words, we could implement "scikits" that are distributed separately but reside in the scipy.* namespace with no further work (for both egg-based and normal distributions). And then there's the question of whether this is a good idea...
I like Fernando's approach, plus 'all' is quicker to type than 'interactive' at an interactive prompt :-) Gary R.
participants (6)
-
Andrew Straw -
Ed Schofield -
Fernando Perez -
Gary Ruben -
Pearu Peterson -
Travis Oliphant