
Often when exploring an object with the 'dir' function, particularly in large packages like SciPy, I find that I need to filter the outout. Since a dir reminds me of a dos 'dir' or linux 'ls', a glob feels like the most natural to use. For example, none of these would work:
dir(sp.fft.i*) # syntax error dir('sp.fft.i*') # returns the attributes a string
I believe a new dir functions is needed, or a change in the behviour of the current version. Of course a 'glob aware dir function' can be implemented by monitoring the call stack (cf. sys._getframe().f_back), but I think the problem is general enough to warrant an official sultion. Sturla

Sturla Molden <sturla@molden.no> writes:
Often when exploring an object with the 'dir' function, particularly in large packages like SciPy, I find that I need to filter the outout.
We have list comprehensions and generator expressions to filter a sequence, and I suspect they would serve you well for this purpose.
What would you expect this to return? The ‘dir’ function is specifically for inspecting *one* object. Do you just want all the attributes of all the matching objects mixed up together, or what?
I believe a new dir functions is needed, or a change in the behviour of the current version.
This is ‘python-ideas’. What is your idea for the desired behaviour? What about using a list comprehension or generator expression to get what you want? -- \ “A learning experience is one of those things that say, “You | `\ know that thing you just did? Don't do that.”” —Douglas Adams, | _o__) 2000-04-05 | Ben Finney

On Thu, Jun 30, 2011 at 8:29 AM, Ben Finney <ben+python@benfinney.id.au> wrote:
The problem is that dir is generally used on a live section, and a list comprehension to filter dir results is way to verbose to type when you need it.
I believe a second parameter to dir, being a glob filter string, would be fine. Being able to type: dir (gtk.Window, "set*") , for example instead of [attrib for attrib in dir(gtk.Window) if attrib.startswith("set")] would be indeed a good thing to have. js -><-

On 30 June 2011 14:14, Sturla Molden <sturla@molden.no> wrote:
Sounds like what you're really after is a richer interactive environment. Have you looked at IPython? (Disclaimer: I don't use it myself, but I know it adds *lots* of features to make the interactive experience better and more productive). Paul.

Den 30.06.2011 14:29, skrev Ben Finney:
We have list comprehensions and generator expressions to filter a sequence, and I suspect they would serve you well for this purpose.
Dir is often used to explore objects or modules interactively, which means we need a very short syntax for this to be handy. Perhaps something like this: dir(object, "foo*") Making an utility function like this with glob or regex is of course extremely simple. Anyone can do it for themselves, how ever they like. What I am asking is if the need to filter the output from dir is so common that it could warrant a change to Python? Sturla

On Thu, Jun 30, 2011 at 11:27 PM, Sturla Molden <sturla@molden.no> wrote:
What I am asking is if the need to filter the output from dir is so common that it could warrant a change to Python?
No, if an object is complicated enough that pprint(dir(obj)) and help(obj) aren't adequate to explore it, then it is time to go read the documentation (or the source, if the documentation is lacking). The interactive prompt is just one of the available tools for code exploration, it doesn't make sense to try to make it do everything. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan wrote:
Do you think that *reading the source* is to be preferred over a simple tool like running a filter over the output of dir()? I'm not exactly sure what point you're trying to make, but I don't think it's a good one.
The interactive prompt is just one of the available tools for code exploration, it doesn't make sense to try to make it do everything.
But the interactive prompt already does everything. Anything you can do in Python can be done at the prompt. The question we are asking is not should users be able to filter the output of dir, because of course they can. The question we are asking is, should there be One Obvious Way to filter the output of dir, rather than the status quo (people do without, reinvent the wheel, or struggle with non-obvious, inconvenient and verbose ways). I'm +1 on adding a glob filter to dir. -- Steven

Steven D'Aprano <steve@pearwood.info> writes:
I understood Nick's meaning to entail that in such a circumstance where usage for a complex object is difficult to discover, the solution isn't to alter Python but to address the poor documentation and/or poor design of those complex objects. But that might be because that's *my* position. -- \ “[F]reedom of speech does not entail freedom to have your ideas | `\ accepted by governments and incorporated into law and policy.” | _o__) —Russell Blackford, 2010-03-06 | Ben Finney

On Fri, Jul 1, 2011 at 10:17 AM, Ben Finney <ben+python@benfinney.id.au> wrote:
Yeah, that's about it - dir-based exploration hits a point of diminishing returns where it becomes better to go find a source of information that doesn't require you to explicitly request each piece of additional data. However, I found Steven and Georg's point about "how do I spell that again?" for objects with a broad API much more compelling, since I *have* been in that situation many, many times. It's not exploration in that case, it's a dir-based equivalent of code completion support. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Sturla Molden <sturla@molden.no> writes:
dir(object, "foo*")
I ask again: what would you expect (give an example) the output of this to be?
What I am asking is if the need to filter the output from dir is so common that it could warrant a change to Python?
Given that we already have ways to filter a sequence built into the language, I doubt the need for a special way to filter the output from some particular function. -- \ “I used to be a proofreader for a skywriting company.” —Steven | `\ Wright | _o__) | Ben Finney

Ben Finney wrote:
If I were to guess:
Example usage: what was the name of the function to remove a directory and all empty parents again?
import os dir(os)
[snip list with more than 200 names]
dir(os, "r*d*") ['read', 'readlink', 'removedirs', 'rmdir']
Ah, I think I remember it now...
What I am asking is if the need to filter the output from dir is so common that it could warrant a change to Python?
At the moment I'm writing list comprehensions in cases like the above, but I'd welcome the addition of a glob or regex argument to dir().

Ben Finney wrote:
That specific example should return an empty list, since object has no attributes that match the glob foo*
There is precedence though. Many string methods have special way to limit their output to some subset of results, rather than relying on a generic, built-in way to do the same thing: mystr.find("spam", 23, 42) vs mystr[23:42].find("spam") dir itself already duplicates functionality available elsewhere:
dir() == sorted(globals()) True
dir with an argument is a little trickier, but it's not far off a one-liner: dir(d) == sorted( set(sum((o.__dict__.keys() for o in ((d,)+type(d).__mro__)), [])) ) (at least for new-style objects with no __slots__). dir is a convenience function, designed for interactive use. The docs make it explicit: [quote] Because dir() is supplied primarily as a convenience for use at an interactive prompt, it tries to supply an interesting set of names more than it tries to supply a rigorously or consistently defined set of names... http://docs.python.org/library/functions.html#dir Given that, I see no downside to making dir more convenient for interactive use. That's what it's for. -- Steven

On 30.06.2011 18:31, Steven D'Aprano wrote:
And don't forget about __dir__...
I agree. I often am looking for a specific member that I know exists, but don't recall the exact name (and in particular, not what the name starts with: at least dir() output is sorted). Searching through one screenful of members isn't pretty. So, +1 for the second argument. Georg

On Fri, Jul 1, 2011 at 3:22 AM, Georg Brandl <g.brandl@gmx.net> wrote:
Yep, I found Steven's reply quite persuasive, so consider my objection withdrawn. And for interactive prompt usage, simple glob-style matching is a better choice than the more powerful re. However, this now needs a tracker issue and a patch - as a builtin that may legitimately be used before the import machinery is fully initialised, it isn't really acceptable for dir() to depend on the fnmatch module just for this feature, so implementing this isn't going to be quite as easy as it might otherwise be. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 01.07.2011 02:12, Nick Coghlan wrote:
Well, apparently someone called "Guido van Rossum" wrote a C implementation of fnmatch once, that is now included in all sorts of software (and in PHP): http://www.google.de/search?q=fnmatch+"guido+van+rossum" <wink> Georg

I would rather add a grep method to regular expression objects: def grep(self,sequence): for s in sequence: if self.search(s): yield s And a global function to the re module: def grep(regex,sequence,flags=0): return re.compile(regex,flags).grep(sequence) Then you could do: re.grep("^i", dir(sp.fft)) Yes, it would be more to type but it would be more general. OT: I think it would be good if regular expressions would be callable with this method: def __call__(self,s): return self.search(s) is not None Then one could use it in a filter expression: filter(re.compile("^i"), dir(sp.fft)) On 06/30/2011 01:38 PM, Sturla Molden wrote:

On 30.06.2011 15:09, Mathias Panzenböck wrote:
No you couldn't, since this would return <generator ...> -- an explicit list() would be required.
What's wrong with filter(re.compile("^i").search, dir(sp.fft)) ? And for non-interactive use, you almost always have a compiled regex object already, so that this becomes filter(rex.search, seq) which is not sufficiently harder than rex.grep(seq) to justify a new regex method. (Also, people won't be able to remember if grep() uses match() or search().) Georg

Georg Brandl <g.brandl@gmx.net> writes:
(I'm confused over this example, since there are no attributes of ‘scipy.fft’ that start with ‘i’. I'll switch to an example where we're looking for its attributes that start with ‘f’.) The globs people have shown are AFAICT easily done by the existing string methods, and don't need regex. So why not:
-- \ “I don't accept the currently fashionable assertion that any | `\ view is automatically as worthy of respect as any equal and | _o__) opposite view.” —Douglas Adams | Ben Finney

On Fri, Jul 1, 2011 at 10:27 AM, Ben Finney <ben+python@benfinney.id.au> wrote:
See Steven's quote about the reason for dir's very existence (i.e. as a convenience function). There's no real comparison from a usability point of view between the above and "dir(scipy.fft, 'f*')" fnmatch.filter(dir(scipy.fft), "f*") is even simpler than the list comprehension, but still no match for the builtin version. However, as I mentioned in my other email, the dependency problem means this is likely to be harder to implement than it first looks, so the next step if for someone that is sufficiently interested to create a patch. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Thu, Jun 30, 2011 at 9:38 PM, Sturla Molden <sturla@molden.no> wrote:
import fnmatch def glob_dir(obj, pattern): return fnmatch.filter(dir(obj), pattern)
print(glob_dir(str, "c*")) ['capitalize', 'center', 'count']
If it's something you use often, drop it into a utility module or PYTHONSTARTUP Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Sturla Molden wrote:
I have added a recipe for this here: http://code.activestate.com/recipes/577774-enhancing-dir-with-globs/ Comments and improvements welcome. -- Steven

Sturla Molden <sturla@molden.no> writes:
Often when exploring an object with the 'dir' function, particularly in large packages like SciPy, I find that I need to filter the outout.
We have list comprehensions and generator expressions to filter a sequence, and I suspect they would serve you well for this purpose.
What would you expect this to return? The ‘dir’ function is specifically for inspecting *one* object. Do you just want all the attributes of all the matching objects mixed up together, or what?
I believe a new dir functions is needed, or a change in the behviour of the current version.
This is ‘python-ideas’. What is your idea for the desired behaviour? What about using a list comprehension or generator expression to get what you want? -- \ “A learning experience is one of those things that say, “You | `\ know that thing you just did? Don't do that.”” —Douglas Adams, | _o__) 2000-04-05 | Ben Finney

On Thu, Jun 30, 2011 at 8:29 AM, Ben Finney <ben+python@benfinney.id.au> wrote:
The problem is that dir is generally used on a live section, and a list comprehension to filter dir results is way to verbose to type when you need it.
I believe a second parameter to dir, being a glob filter string, would be fine. Being able to type: dir (gtk.Window, "set*") , for example instead of [attrib for attrib in dir(gtk.Window) if attrib.startswith("set")] would be indeed a good thing to have. js -><-

On 30 June 2011 14:14, Sturla Molden <sturla@molden.no> wrote:
Sounds like what you're really after is a richer interactive environment. Have you looked at IPython? (Disclaimer: I don't use it myself, but I know it adds *lots* of features to make the interactive experience better and more productive). Paul.

Den 30.06.2011 14:29, skrev Ben Finney:
We have list comprehensions and generator expressions to filter a sequence, and I suspect they would serve you well for this purpose.
Dir is often used to explore objects or modules interactively, which means we need a very short syntax for this to be handy. Perhaps something like this: dir(object, "foo*") Making an utility function like this with glob or regex is of course extremely simple. Anyone can do it for themselves, how ever they like. What I am asking is if the need to filter the output from dir is so common that it could warrant a change to Python? Sturla

On Thu, Jun 30, 2011 at 11:27 PM, Sturla Molden <sturla@molden.no> wrote:
What I am asking is if the need to filter the output from dir is so common that it could warrant a change to Python?
No, if an object is complicated enough that pprint(dir(obj)) and help(obj) aren't adequate to explore it, then it is time to go read the documentation (or the source, if the documentation is lacking). The interactive prompt is just one of the available tools for code exploration, it doesn't make sense to try to make it do everything. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan wrote:
Do you think that *reading the source* is to be preferred over a simple tool like running a filter over the output of dir()? I'm not exactly sure what point you're trying to make, but I don't think it's a good one.
The interactive prompt is just one of the available tools for code exploration, it doesn't make sense to try to make it do everything.
But the interactive prompt already does everything. Anything you can do in Python can be done at the prompt. The question we are asking is not should users be able to filter the output of dir, because of course they can. The question we are asking is, should there be One Obvious Way to filter the output of dir, rather than the status quo (people do without, reinvent the wheel, or struggle with non-obvious, inconvenient and verbose ways). I'm +1 on adding a glob filter to dir. -- Steven

Steven D'Aprano <steve@pearwood.info> writes:
I understood Nick's meaning to entail that in such a circumstance where usage for a complex object is difficult to discover, the solution isn't to alter Python but to address the poor documentation and/or poor design of those complex objects. But that might be because that's *my* position. -- \ “[F]reedom of speech does not entail freedom to have your ideas | `\ accepted by governments and incorporated into law and policy.” | _o__) —Russell Blackford, 2010-03-06 | Ben Finney

On Fri, Jul 1, 2011 at 10:17 AM, Ben Finney <ben+python@benfinney.id.au> wrote:
Yeah, that's about it - dir-based exploration hits a point of diminishing returns where it becomes better to go find a source of information that doesn't require you to explicitly request each piece of additional data. However, I found Steven and Georg's point about "how do I spell that again?" for objects with a broad API much more compelling, since I *have* been in that situation many, many times. It's not exploration in that case, it's a dir-based equivalent of code completion support. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Sturla Molden <sturla@molden.no> writes:
dir(object, "foo*")
I ask again: what would you expect (give an example) the output of this to be?
What I am asking is if the need to filter the output from dir is so common that it could warrant a change to Python?
Given that we already have ways to filter a sequence built into the language, I doubt the need for a special way to filter the output from some particular function. -- \ “I used to be a proofreader for a skywriting company.” —Steven | `\ Wright | _o__) | Ben Finney

Ben Finney wrote:
If I were to guess:
Example usage: what was the name of the function to remove a directory and all empty parents again?
import os dir(os)
[snip list with more than 200 names]
dir(os, "r*d*") ['read', 'readlink', 'removedirs', 'rmdir']
Ah, I think I remember it now...
What I am asking is if the need to filter the output from dir is so common that it could warrant a change to Python?
At the moment I'm writing list comprehensions in cases like the above, but I'd welcome the addition of a glob or regex argument to dir().

Ben Finney wrote:
That specific example should return an empty list, since object has no attributes that match the glob foo*
There is precedence though. Many string methods have special way to limit their output to some subset of results, rather than relying on a generic, built-in way to do the same thing: mystr.find("spam", 23, 42) vs mystr[23:42].find("spam") dir itself already duplicates functionality available elsewhere:
dir() == sorted(globals()) True
dir with an argument is a little trickier, but it's not far off a one-liner: dir(d) == sorted( set(sum((o.__dict__.keys() for o in ((d,)+type(d).__mro__)), [])) ) (at least for new-style objects with no __slots__). dir is a convenience function, designed for interactive use. The docs make it explicit: [quote] Because dir() is supplied primarily as a convenience for use at an interactive prompt, it tries to supply an interesting set of names more than it tries to supply a rigorously or consistently defined set of names... http://docs.python.org/library/functions.html#dir Given that, I see no downside to making dir more convenient for interactive use. That's what it's for. -- Steven

On 30.06.2011 18:31, Steven D'Aprano wrote:
And don't forget about __dir__...
I agree. I often am looking for a specific member that I know exists, but don't recall the exact name (and in particular, not what the name starts with: at least dir() output is sorted). Searching through one screenful of members isn't pretty. So, +1 for the second argument. Georg

On Fri, Jul 1, 2011 at 3:22 AM, Georg Brandl <g.brandl@gmx.net> wrote:
Yep, I found Steven's reply quite persuasive, so consider my objection withdrawn. And for interactive prompt usage, simple glob-style matching is a better choice than the more powerful re. However, this now needs a tracker issue and a patch - as a builtin that may legitimately be used before the import machinery is fully initialised, it isn't really acceptable for dir() to depend on the fnmatch module just for this feature, so implementing this isn't going to be quite as easy as it might otherwise be. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 01.07.2011 02:12, Nick Coghlan wrote:
Well, apparently someone called "Guido van Rossum" wrote a C implementation of fnmatch once, that is now included in all sorts of software (and in PHP): http://www.google.de/search?q=fnmatch+"guido+van+rossum" <wink> Georg

I would rather add a grep method to regular expression objects: def grep(self,sequence): for s in sequence: if self.search(s): yield s And a global function to the re module: def grep(regex,sequence,flags=0): return re.compile(regex,flags).grep(sequence) Then you could do: re.grep("^i", dir(sp.fft)) Yes, it would be more to type but it would be more general. OT: I think it would be good if regular expressions would be callable with this method: def __call__(self,s): return self.search(s) is not None Then one could use it in a filter expression: filter(re.compile("^i"), dir(sp.fft)) On 06/30/2011 01:38 PM, Sturla Molden wrote:

On 30.06.2011 15:09, Mathias Panzenböck wrote:
No you couldn't, since this would return <generator ...> -- an explicit list() would be required.
What's wrong with filter(re.compile("^i").search, dir(sp.fft)) ? And for non-interactive use, you almost always have a compiled regex object already, so that this becomes filter(rex.search, seq) which is not sufficiently harder than rex.grep(seq) to justify a new regex method. (Also, people won't be able to remember if grep() uses match() or search().) Georg

Georg Brandl <g.brandl@gmx.net> writes:
(I'm confused over this example, since there are no attributes of ‘scipy.fft’ that start with ‘i’. I'll switch to an example where we're looking for its attributes that start with ‘f’.) The globs people have shown are AFAICT easily done by the existing string methods, and don't need regex. So why not:
-- \ “I don't accept the currently fashionable assertion that any | `\ view is automatically as worthy of respect as any equal and | _o__) opposite view.” —Douglas Adams | Ben Finney

On Fri, Jul 1, 2011 at 10:27 AM, Ben Finney <ben+python@benfinney.id.au> wrote:
See Steven's quote about the reason for dir's very existence (i.e. as a convenience function). There's no real comparison from a usability point of view between the above and "dir(scipy.fft, 'f*')" fnmatch.filter(dir(scipy.fft), "f*") is even simpler than the list comprehension, but still no match for the builtin version. However, as I mentioned in my other email, the dependency problem means this is likely to be harder to implement than it first looks, so the next step if for someone that is sufficiently interested to create a patch. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Thu, Jun 30, 2011 at 9:38 PM, Sturla Molden <sturla@molden.no> wrote:
import fnmatch def glob_dir(obj, pattern): return fnmatch.filter(dir(obj), pattern)
print(glob_dir(str, "c*")) ['capitalize', 'center', 'count']
If it's something you use often, drop it into a utility module or PYTHONSTARTUP Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Sturla Molden wrote:
I have added a recipe for this here: http://code.activestate.com/recipes/577774-enhancing-dir-with-globs/ Comments and improvements welcome. -- Steven
participants (13)
-
Ben Finney
-
Georg Brandl
-
Joao S. O. Bueno
-
Masklinn
-
Mathias Panzenböck
-
Mike Graham
-
Nick Coghlan
-
Paul Moore
-
Peter Otten
-
Stefan Behnel
-
Steven D'Aprano
-
Sturla Molden
-
Terry Reedy