--- Guido van Rossum wrote:

On 6/21/06, Ralf W. Grosse-Kunstleve wrote:

...

I am getting tons of "ImportWarning: Not importing directory". See below for examples. It is impractical for me to reorganize our directory structure. I'd be busy for a week or more and people would probably scream at me because all the paths have changed. Are there other options to get rid of the warnings?

Check out the -W command line option and the warnings module. These document how to suppress warnings.

Thanks! This does the trick: python -W'ignore:Not importing directory' But this doesn't: python -W'ignore:Not importing directory:ImportWarning' I tried a bunch of variations without success. A few examples here would be very valuable: http://docs.python.org/lib/warning-filter.html Also, the magic incantation to silence the warnings would be very helpful here: http://docs.python.org/dev/whatsnew/other-lang.html Is there a way to set the warning options via an environment variable? Otherwise I am forced to use a wrapper or aliases. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

--- "Martin v. Löwis" wrote:

Ralf W. Grosse-Kunstleve wrote:

...

Is there a way to set the warning options via an environment variable?

This is off-topic for python-dev,

What is the channel I should use? (I am testing a beta 1.)

but: why don't switch off the warnings in the code?

We support installation from sources with the native Python if available. Any Python >= 2.2.1 works. It would be frustrating if we had to give up on this just because of a warning designed for newcomers. In our applications we typically address this type of problem with informative exceptions. For example, if a Boost.Python wrapped C++ object doesn't support pickling: RuntimeError: Pickling of "cctbx_sgtbx_ext.space_group_symbols" instances is not enabled (http://www.boost.org/libs/python/doc/v2/pickle.html) Something like this could help newcomers just the same without impacting experienced users with large, complex package structures. E.g.:

...

...

import mypackage.foo Traceback (most recent call last): File "<stdin>", line 1, in ? ImportError: No module named mypackage.foo Note that subdirectories are searched for imports only if they contain an __init__.py file: http://www.python.org/doc/essays/packages.html

__________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

--- "Martin v. L�wis" wrote:

The specific question was

"Is there a way to set the warning options via an environment variable?"

This has nothing to do with beta1; the warnings module was introduced many releases ago, along with all the mechanics to disable warnings.

Due to the new ImportWarning first introduced in 2.5b1 the question of disabling warnings is becoming much more pressing (I am assuming that I am not again the only person on the planet to have this problem).

I guess you misunderstood.

Yes.

I propose you put warnings.simplefilter() into your code. The warnings was introduced before 2.2.1 IIRC, so this should work on all releases you want to support (but have no effect on installations where the warning isn't generated).

Where would I put the warnings.simplefilter()? I have hundreds of scripts and __init__.py files. I just came accross this situation (simplified): % cd boost/boost % python2.5

...

...

import math __main__:1: ImportWarning: Not importing directory 'math': missing __init__.py

This is because there is a subdirectory math in boost/boost, something that I cannot change. The PYTHONPATH is not set at all in this case. I.e. I get the ImportWarning just because my current working directory happens to contain a subdirectory which matches one of the Python modules in the standard library. Isn't this going to cause widespread problems? __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

--- "Martin v. Löwis" wrote:

I don't know. Whether a warning is a problem is a matter of attitude, also.

Our users will think our applications are broken if they see warnings like that. It is not professional. __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

On Sat, 24 Jun 2006 11:47:19 +0200, "\"Martin v. Löwis\"" wrote:

Ralf W. Grosse-Kunstleve wrote:

...

--- "Martin v. Löwis" wrote:

...

I don't know. Whether a warning is a problem is a matter of attitude, also.

Our users will think our applications are broken if they see warnings like that. It is not professional.

Actually, your application *was* pretty close to being broken a few weeks ago, when Guido wanted to drop the requirement that a package must contain an __init__ file. In that case, "import math" would have imported the directory, and given you an empty package.

But this change was *not* made, and afaict it is not going to be made. So the application is not broken, and the warning is entirely spurious. I am very unhappy that the burden of understanding Python's package structure is being pushed onto end users in this way. Several of my projects now emit three or four warnings on import now. The Twisted plugin system relies on the fact that directories without __init__ are not Python packages (since they _aren't_, have never been, and it has always been extremely clear that Python will ignore them). Of course, Twisted is a pretty marginal Python user so I'm sure no one cares.

On Sat, 24 Jun 2006 16:24:11 +0200, "\"Martin v. Löwis\"" wrote:

Jean-Paul Calderone wrote:

...

I am very unhappy that the burden of understanding Python's package structure is being pushed onto end users in this way. Several of my projects now emit three or four warnings on import now.

So are you requesting that the change is reverted?

Yes, please.

Regards, Martin

On Sat, 24 Jun 2006 07:27:15 -0700, Aahz wrote:

On Sat, Jun 24, 2006, Jean-Paul Calderone wrote:

...

I am very unhappy that the burden of understanding Python's package structure is being pushed onto end users in this way. Several of my projects now emit three or four warnings on import now.

The Twisted plugin system relies on the fact that directories without __init__ are not Python packages (since they _aren't_, have never been, and it has always been extremely clear that Python will ignore them).

Of course, Twisted is a pretty marginal Python user so I'm sure no one cares.

Then again, bringing this back to the original source of this change, Google is a pretty marginal Python user, too. ;-)

I think it is safe to say that Twisted is more widely used than anything Google has yet released. Twisted also has a reasonably plausible technical reason to dislike this change. Google has a bunch of engineers who, apparently, cannot remember to create an empty __init__.py file in some directories sometimes.

I was a pretty strong -1 on the original proposed change of allowing import on empty directories, but my take is that if a project deliberately includes empty directories, they can add a new warning filter on program startup. Your users will have to upgrade to a new version of the application or do a similar fix in their own sitecustomize. I don't consider that a huge burden.

The usage here precludes fixing it in Twisted. Importing twisted itself prints a warning: there's no way to get code run soon enough to suppress this. I do think requiring each user to modify sitecustomize is overly burdensome. Of course this is highly subjective and I don't expect anyone to come to an agreement over it, but it seems clear that it is at least a burden of some sort. Jean-Paul

On Jun 24, 2006, at 1:29 PM, Ralf W. Grosse-Kunstleve wrote:

--- Jean-Paul Calderone wrote:

...

I think it is safe to say that Twisted is more widely used than anything Google has yet released. Twisted also has a reasonably plausible technical reason to dislike this change. Google has a bunch of engineers who, apparently, cannot remember to create an empty __init__.py file in some directories sometimes.

Simply adding a note to the ImportError message would solve this problem "just in time":

...

...

...

import mypackage.foo Traceback (most recent call last): File "<stdin>", line 1, in ? ImportError: No module named mypackage.foo Note that subdirectories are searched for imports only if they contain an __init__.py file: http://www.python.org/doc/essays/packages.html

I also dislike the warning solution. Making the ImportError message more verbose seems like a much nicer solution. James

On Jun 25, 2006, at 9:47 PM, James Y Knight wrote:

On Jun 24, 2006, at 1:29 PM, Ralf W. Grosse-Kunstleve wrote:

...

--- Jean-Paul Calderone wrote:

...

I think it is safe to say that Twisted is more widely used than anything Google has yet released. Twisted also has a reasonably plausible technical reason to dislike this change. Google has a bunch of engineers who, apparently, cannot remember to create an empty __init__.py file in some directories sometimes.

Simply adding a note to the ImportError message would solve this problem "just in time":

...

...

...

import mypackage.foo Traceback (most recent call last): File "<stdin>", line 1, in ? ImportError: No module named mypackage.foo Note that subdirectories are searched for imports only if they contain an __init__.py file: http://www.python.org/doc/essays/packages.html

I also dislike the warning solution. Making the ImportError message more verbose seems like a much nicer solution.

I just found another reason to dislike the warnings: my homedir on one machine has a lot of random directories in it. One of them is named "readline". Every time I run python 2.5, it now helpfully notes: sys:1: ImportWarning: Not importing directory 'readline': missing __init__.py It used to be the case that it was very unlikely that running python in your homedir would cause issues. Even though the current directory is on the default pythonpath, you needed to have either a file ending in .py or a directory with an __init__.py with the same name as a python module to cause problems. And that is generally unlikely to happen. Now, however, you get warnings just by having _any_ directory in your CWD with the same name as a python module. That's much more likely to happen; I can't be the only one who will have this issue. I'd like to suggest the simple solution quoted above with a constant string added to the ImportError message would be good enough, and better than the current warning situation. Clearly it would be even better if someone did the complicated thing of keeping track of which directories would have been used had they had __init__.py files in them, and appending _that_ to the eventual ImportError message, but I don't think removing the warning should be held up on doing that. James

Ralf W. Grosse-Kunstleve wrote:

--- Jean-Paul Calderone wrote:

...

I think it is safe to say that Twisted is more widely used than anything Google has yet released. Twisted also has a reasonably plausible technical reason to dislike this change. Google has a bunch of engineers who, apparently, cannot remember to create an empty __init__.py file in some directories sometimes.

Simply adding a note to the ImportError message would solve this problem "just in time":

...

...

...

import mypackage.foo

Traceback (most recent call last): File "<stdin>", line 1, in ? ImportError: No module named mypackage.foo Note that subdirectories are searched for imports only if they contain an __init__.py file: http://www.python.org/doc/essays/packages.html

Yeah, that'll really help the end-user whose sys admin has just upgraded to 2.5, won't it? regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Love me, love my blog http://holdenweb.blogspot.com Recent Ramblings http://del.icio.us/steve.holden

Guido van Rossum wrote:

On 6/24/06, Jean-Paul Calderone wrote:

...

...

Actually, your application *was* pretty close to being broken a few weeks ago, when Guido wanted to drop the requirement that a package must contain an __init__ file. In that case, "import math" would have imported the directory, and given you an empty package. But this change was *not* made, and afaict it is not going to be made.

Correct. We'll stick with the warning. (At least until Py3k but most likely also in Py3k.)

Perhaps ImportWarning should default to being ignored, the same way PendingDeprecationWarning does? Then -Wd would become 'the one obvious way' to debug import problems, since it would switch ImportWarning on without drowning you in a flood of import diagnostics the way -v can do. Import Errors could even point you in the right direction:

...

...

import mypackage.foo Traceback (most recent call last): File "<stdin>", line 1, in ? ImportError: No module named mypackage.foo Diagnostic import warnings can be enabled with -Wd

Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

Benji York writes:

Nick Coghlan wrote:

...

Perhaps ImportWarning should default to being ignored, the same way PendingDeprecationWarning does?

Then -Wd would become 'the one obvious way' to debug import problems

+1

I'm not sure what this would achieve -- people who don't know enough about Python to add __init__.py files aren't going to know enough to make suppressed-by-default warnings not suppressed. The more I think about it, the more I like the idea of saying something when an import fails only because of a missing __init__.py file. I guess I should try to implement it... Cheers, mwh -- I also feel it essential to note, [...], that Description Logics, non-Monotonic Logics, Default Logics and Circumscription Logics can all collectively go suck a cow. Thank you. -- http://advogato.org/person/Johnath/diary.html?start=4

On Sun, 25 Jun 2006 17:51:17 -0700, Guido van Rossum wrote:

On 6/24/06, Jean-Paul Calderone wrote:

...

...

Actually, your application *was* pretty close to being broken a few weeks ago, when Guido wanted to drop the requirement that a package must contain an __init__ file. In that case, "import math" would have imported the directory, and given you an empty package.

But this change was *not* made, and afaict it is not going to be made.

Correct. We'll stick with the warning. (At least until Py3k but most likely also in Py3k.)

Even given that it emits completely spurious warnings for any package that happens to share a name with a directory in whatever the working path is (say, your home directory)? How about if someone grovels through import.c and figures out how to make the warning information only show up if the import actually fails? Jean-Paul

Martin v. Löwis wrote:

Actually, your application *was* pretty close to being broken a few weeks ago, when Guido wanted to drop the requirement that a package must contain an __init__ file.

BTW, when that was being discussed, did anyone consider allowing a directory to be given a .py suffix as an alternative way to mark it as a package? -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiem! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing@canterbury.ac.nz +--------------------------------------+

On 6/25/06, Fred L. Drake, Jr. wrote:

On Sunday 25 June 2006 20:48, Greg Ewing wrote:

...

BTW, when that was being discussed, did anyone consider allowing a directory to be given a .py suffix as an alternative way to mark it as a package? :-) I'd certainly be a lot happier with that than with the current behavior. Silly little warnings about perfectly good data-only directories are just silly.

And silly whining about warnings for silly name conflicts are just as silly. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Ralf W. Grosse-Kunstleve wrote:

I can work around it, sure. Everybody can work around it, of course. But consider that one hour of a professional person is at least $100 with benefits etc. included. (If that sounds high, I know people charging much more than that; also consider that the going rate for a car mechanic in the bay area is $90, as you probably know.) Now say you have 1000 groups of developers having to work around the warning (I bet you have more). There will be discussions, alternatives will be tried and discarded, etc. Say that eats about 10 man hours per group before the issue is settled, which again is a very conservative estimate I believe. That makes a total of $1000000 in damages, at least. Is that warning really worth a millon dollar?

So spend some of the money to come up with an alternate solution for 2.5b2. With a potential damage of a million dollars, it shouldn't be too difficult to provide a patch by tomorrow, right? This is open source, folks. Business arguments don't matter much to many of us. I don't get any money for my contributions to Python, and I'm not whining about all the lost consultant fees I could have collected while contributing to Python instead. What matters are actual contributions: bug reports, patches, PEPs, etc. In the specific case, make sure that your alternative solution not only makes you happy, but also solves the original problem as good or better than the current solution (read some email archives to find out what the original problem was). Regards, Martin

Ralf W. Grosse-Kunstleve wrote:

If there is a consenus, I'd create a new exception ImportErrorNoModule(name) that is used consistently from all places. This would ensure uniformity of the message in the future.

A correction proposal should only be given if it is likely correct. There can be many reasons why an import could fail: there might be no read permission for the file, or the PYTHONPATH might be setup incorrectly. IOW, a hint about a missing __init__.py should only be given if a directory with the name of module was found, but lacked an __init__.py (i.e. in the cases where currently a warning is produced). Regards, Martin

Ralf W. Grosse-Kunstleve wrote:

...

There can be many reasons why an import could fail: there might be no read permission for the file,

The warning in 2.5b1 doesn't fire in this case:

Sure, but it would produce your "note", right? And the note would be essentially wrong. Instead, the ImportError should read ImportError: No module named junk; could not open junk.py (permission denied)

...

or the PYTHONPATH might be setup incorrectly.

That's impossible to detect.

Right. So the ImportError should not guess that there is a problem with packages if there could be dozens of other reasons why the import failed.

I am thinking you'd need to build up a buffer of potential warnings while trying to resolve an import. If the import succeeds the buffer is discarded, if it fails it is added to the exception message, or the warnings are "flushed" right before the ImportError is raised. Does that sound right?

That might work, yes.

How would this interact with threading (it seems you'd need a separate buffer for each thread)?

There are several solutions. I think you are holding the import lock all the time, so there can be only one import running (one would have to check whether the import lock is really held all the time); in that case, a global variable would work just fine. Another option is to pass-through all import-related data across all function calls as a parameter; that may actually cause a reduction in the number of parameters to the current functions, and simplify the code. Define a struct to hold all the relevant data, allocate it when entering the import code, pass it to every function, fill it out as needed, and deallocate it when leaving the import code. Allocation of the struct itself could likely be done on stack. Yet another option is to put the data into thread storage (although care is needed wrt. recursive imports within one thread). Regards, Martin

On Sat, Jul 01, 2006, Sergey A. Lipnevich wrote:

Please let me know if this would work and if anything needs to be done for this patch to be accepted.

The first thing you need to do for ANY patch to be considered is to post it so SourceForge (or at least post to python-dev explaining that you'll post to SF as soon as it comes back). -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I saw `cout' being shifted "Hello world" times to the left and stopped right there." --Steve Gonedes

Martin v. Löwis wrote:

Sergey A. Lipnevich wrote:

...

I tried to implement Jean-Paul Calderone's idea for the following patch, plagiarizing Ralf W. Grosse-Kunstleve's error text. It delays import ... Please notice that there is also python.org/sf/1515361

I had no time to compare this with your patch, yet.

Thanks! I made python.org/sf/1515609. The difference (also documented in the description) is that my patch saves memory and some CPU cycles by not trying to collect all directories Python did not import because of missing __init__.py. It only reports how many such directories there are and what is the first one. Sergey.

A.M. Kuchling wrote:

On Mon, Jun 26, 2006 at 08:29:49AM +0200, "Martin v. Löwis" wrote:

...

(read some email archives to find out what the original problem was).

People at Google don't read manuals?

The documentation of how imports actually work isn't that easy to find? Guido's package essay on python.org and PEP 302 seem to cover the topic pretty well between them, but neither of them is part of the normal documentation. The situation shares some unfortunate similarities with that of the new-style class documentation - it's documented, just not in the places you might initially expect :( Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org