Mailman 3 April 2006 - Python-Dev

PY_FORMAT_SIZE_T warnings on OS X
by Brett Cannon 27 Apr '06

27 Apr '06

I think these are all Tim's fault =) : Objects/object.c: In function '_Py_NegativeRefcount': Objects/object.c:144: warning: format '%d' expects type 'int', but argument 7 has type 'Py_ssize_t' Objects/stringobject.c: In function 'PyString_FromFormatV': Objects/stringobject.c:278: warning: format '%u' expects type 'unsigned int', but argument 3 has type 'size_t' Python/pythonrun.c: In function 'Py_Finalize': Python/pythonrun.c:393: warning: format '%d' expects type 'int', but argument 3 has type 'Py_ssize_t' Python/pythonrun.c: In function 'PyRun_InteractiveLoopFlags': Python/pythonrun.c:683: warning: format '%d' expects type 'int', but argument 3 has type 'Py_ssize_t' Modules/gcmodule.c: In function 'collect': Modules/gcmodule.c:746: warning: format '%d' expects type 'int', but argument 2 has type 'Py_ssize_t' Modules/gcmodule.c:841: warning: format '%d' expects type 'int', but argument 2 has type 'Py_ssize_t' Modules/gcmodule.c:841: warning: format '%d' expects type 'int', but argument 3 has type 'Py_ssize_t' What's odd about them is that that the use of PY_FORMAT_SIZE_T seems to be correct, so I don't know what the problem is. Unfortunately ``gcc -E Include/pyport.h`` is no help since PY_FORMAT_SIZE_T is not showing up defined (possibly because gcc says it can't find pyconfig.h). -Brett

3 9

need info for externally maintained modules PEP
by Brett Cannon 27 Apr '06

27 Apr '06

OK, I am going to write the PEP I proposed a week or so ago, listing all modules and packages within the stdlib that are maintained externally so we have a central place to go for contact info or where to report bugs on issues. This should only apply to modules that want bugs reported outside of the Python tracker and have a separate dev track. People who just use the Python repository as their mainline version can just be left out. For each package I need the name of the module, the name of the maintainer, homepage of the module outside of Python, and where to report bugs. Do people think we need to document the version that has been imported into Python and when that was done as well? Anyway, here is a list of the packages that I think have outside maintenance (or at least have been at some point). Anyone who has info on them that I need, please let me know the details. Also, if I missed any, obviously speak up: - bsddb (still external?) - sqlite - ctypes - cjkcodecs (still external?) - expat - email - logging (still external?) - Tix - ElementTree

12 20

Re: [Python-Dev] Dropping __init__.py requirement for subpackages
by Phillip J. Eby 27 Apr '06

27 Apr '06

At 11:50 AM 4/26/2006 -0700, Guido van Rossum wrote: >I'm not sure what you mean by "one directory read". You'd have to list >the entire directory, which may require reading more than one block if >the directory is large. You have to do this to find an __init__.py too, don't you? Technically, there's going to be a search for a .pyc or .pyo first, anyway. I'm just saying you can stop as soon as you hit an extension that's in imp.get_suffixes(). > > Are you sure you wouldn't rather just write a GoogleImporter class to fix > > this problem? > >No, because that would require more setup code with a requirement to >properly enable it, etc., etc., more failure modes, etc., etc. I don't understand. I thought you said you had only *one* top-level package. Fix *that* package, by putting the code in its __init__.py. Job done. > > Append it to sys.path_hooks, clear sys.path_importer_cache, > > and you're all set. For that matter, if you have only one top-level > > package, put the class and the installation code in that top-level > > __init__, and you're set to go. > >I wish it were that easy. Well, if there's more than one top-level package, put the code in a module called google_imports and "import google_import" in each top-level package's __init__.py. I'm not sure I understand why a solution that works with released versions of Python that allows you to do exactly what you want, is inferior to a hypothetical solution for an unreleased version of Python that forces everybody else to update their tools. Unless of course the problem you're trying to solve is a political one rather than a technical one, that is. Or perhaps it wasn't clear from my explanation that my proposal will work the way you need it to, or I misunderstand what you're trying to do. Anyway, I'm not opposed to the idea of supporting this in future Pythons, but I definitely think it falls under the "but sometimes never is better than RIGHT now" rule where 2.5 is concerned. :) In particular, I'm worried that you're shrugging off the extent of the collateral damage here, and I'd be happiest if we waited until 3.0 before changing this particular rule -- and if we changed it in favor of namespace packages, which will more closely match naive user expectations. However, the "fix the tools" argument is weak, IMO. Zipfile imports have been a fairly half-assed feature for 2.3 and 2.4 because nobody took the time to make the *rest* of the stdlib work with zip imports. It's not a good idea to change machinery like this without considering at least what's going to have to be fixed in the stdlib. At a minimum, pydoc and distutils have embedded assumptions regarding __init__ modules, and I wouldn't be surprised if ihooks, imputil, and others do as well. If we can't keep the stdlib up to date with changes in the language, how can we expect anybody else to keep their code up to date? Finally, as others have pointed out, requiring __init__ at the top level just means that this isn't going to help anybody but Google. ISTM that in most cases outside Google, Python newbies are more likely to be creating top-level packages *first*, so the implicit __init__ doesn't help them. So, to summarize: 1. It only really helps Google 2. It inconveniences others who have to update their tools in order to support people who end up using it (even if by accident) 3. It's not a small change, unless you leave the rest of the stdlib unreviewed for impacts 4. It could be fixed at Google by adding a very small amount of code to the top of your __init__.py files (although apparently this is prevented for mysterious reasons that can't be shared) What's not to like? ;)

3 2

Re: [Python-Dev] Dropping __init__.py requirement for subpackages
by Phillip J. Eby 27 Apr '06

27 Apr '06

At 04:57 PM 4/26/2006 -0700, Guido van Rossum wrote: >On 4/26/06, Delaney, Timothy (Tim) <tdelaney(a)avaya.com> wrote: > > Possibly. Perhaps it would be useful to have `is_package(dirname)`, > > `is_rootpackage(dirname)` and `is_subpackage(dirname)` functions > > somewhere (pkgutils?). > >YAGNI. Also note that not all modules or packages are represented by >pathnames -- they could live in zip files, or be accessed via whatever >other magic an import handler users. FYI, pkgutil in 2.5 has utilities to walk a package tree, starting from sys.path or a package __path__, and it's PEP 302 compliant. pydoc now uses this in place of directory inspection, so that documenting zipped packages works correctly. These functions aren't documented yet, though, and probably won't be until next week at the earliest.

1 0

Re: [Python-Dev] Dropping __init__.py requirement for subpackages
by Phillip J. Eby 26 Apr '06

26 Apr '06

At 01:10 AM 4/27/2006 +0200, Thomas Wouters wrote: >On 4/27/06, Guido van Rossum <<mailto:guido@python.org>guido(a)python.org> >wrote: >>I'd worry that it'll cause complaints when the warning is incorrect >>and a certain directory is being skipped intentionally. E.g. the >>"string" directory that someone had. Getting a warning like this can >>be just as upsetting to newbies! > >I don't think getting a spurious warning is as upsetting as getting no >warning but the damned thing just not working. At least you have something >to google for. And the warning includes the original line of source that >triggered it *and* the directory (or directories) it's complaining about, >which is quite a lot of helpful hints. +1. If the warning is off-base, you can rename the directory or suppress the warning. As for the newbie situation, ISTM that this warning will generally come close enough in time to an environment change (new path entry, newly created conflicting directory) to be seen as informative. The only time it might be confusing is if you had just added an "import foo" after having a "foo" directory sitting around for a while. But even then, the warning is saying, "hey, it looked like you might have meant *this* foo directory... if so, you're missing an __init__". So, at that point I rename the directory... or maybe add the __init__ and break my code. So then I back it out and put up with the warning and complain to c.l.p, or maybe threaten Guido with a pitchfork if I work at Google. Or maybe just a regular-sized fork, since the warning is just annoying. :) >Alrighty then. The list has about 12 hours to convince me (and you) that >it's a bad idea to generate that warning. I'll be asleep by the time the >trunk un-freezes, and I have a string of early meetings tomorrow. I'll get >to it somewhere in the afternoon :) I like the patch in general, but may I suggest PackageWarning or maybe BrokenPackageWarning instead of ImportWarning as the class name?

1 0

Re: [Python-Dev] Dropping __init__.py requirement for subpackages
by Delaney, Timothy (Tim) 26 Apr '06

26 Apr '06

Guido van Rossum wrote: > The difference is that if you find a valid module package later on the > path, you'll still get warnings. This is the bit I don't like about it. Currently the warnings are displayed as the packages are found. I'd be quite happy with the warnings if they were suppressed until there is an actual ImportError - at which time they'll be displayed. > But occasionally those will be useful > too -- when you are trying to create a package that happens to have > the same name as a standard library package or module, it's a lot > easier to debug the missing __init__.py if you get a warning about it > instead of having to figure out that the foo you imported is not the > foo you intended. I suspect that more often any warnings when there is a successful import will be spurious. But testing in the field will reveal that. So I say alpha 3 should have Thomas' patch as is, and it can be changed afterwards if necessary. > Thomas' patch does this automatically -- you get a warning for each > directory that is weighed and found too light. Not exactly - you'll get warnings for directories earlier in sys.path than a matching package. Potential packages later in the path won't be warned about. If you're trying to resolve import problems, it's just as likely that the package you really want is later in sys.path than earlier. Obviously in the case that you get an ImportError this goes away. > Is it also useful to distinguish a subpackage from a non-subpackage? Possibly. Perhaps it would be useful to have `is_package(dirname)`, `is_rootpackage(dirname)` and `is_subpackage(dirname)` functions somewhere (pkgutils?). Then the tools objections go away, and whatever mechanism is necessary to determine this (e.g. is_rootpackage checks if the directory is importable via sys.path) can be implemented. > Anyway, the warning is more compatible and just as helpful so we'll go > with that. Fair enough. Tim Delaney

2 1

Re: [Python-Dev] Dropping __init__.py requirement for subpackages
by Delaney, Timothy (Tim) 26 Apr '06

26 Apr '06

Guido van Rossum wrote: >> http://python.org/sf/1477281 >> >> (You can call it 'oldtimer-repellant' if you want to use it to >> convince people there isn't any *real* backward-compatibility issue.) > > I'd worry that it'll cause complaints when the warning is incorrect > and a certain directory is being skipped intentionally. E.g. the > "string" directory that someone had. Getting a warning like this can > be just as upsetting to newbies! I really think it would be more useful having an ImportError containing a suggestion than having a warning. Anyone who knows it's bogus can just ignore it. I'd probably make it that all directories that could have been imports get listed. FWIW I was definitely a kneejerk -1. After reading all the messages in this thread, I think I'm now a non-kneejerk -1. It seems like gratuitous change introducing inconsistency for minimal benefit - esp. if there is a notification that a directory *could* have been a package on import failure. I think it's a useful feature of Python that it's simple to distinguish a package from a non-package. Tim Delaney

2 1

Re: [Python-Dev] Dropping __init__.py requirement for subpackages
by Phillip J. Eby 26 Apr '06

26 Apr '06

At 01:49 PM 4/26/2006 -0700, Guido van Rossum wrote: >OK, forget it. I'll face the pitchforks. > >I'm disappointed though -- it sounds like we can never change anything >about Python any more because it will upset the oldtimers. I know exactly how you feel. :) But there's always Python 3.0, and if we're refactoring the import machinery there, we can do this the right way, not just the "right now" way. ;) IMO, if Py3K does this, it can and should be inclusive of top-level packages and assemble __path__ using all the sys.path entries. If we're going to break it, let's break it all the way. :) I'm still really curious why the importer solution (especially if tucked away in a Google-defined sitecustomize) won't work, though.

2 1

Re: [Python-Dev] Dropping __init__.py requirement for subpackages
by Terry Jones 26 Apr '06

26 Apr '06

It might be helpful to consider how people would tackle Guido's problem by pretending that a regular Joe (i.e., someone who couldn't consider changing the semantics of Python itself) had asked this question. I would suggest adding a hook to their version control system to automatically create (and preferably also check out) an __init__.py file whenever a new (source code) directory was placed under version control (supposing you can distinguish source code directories from the check in dirname). From one point of view this is a file system issue, so a file system solution might be a good way to solve it. This approach would also allow you to add this behavior/support on a per-user basis. Terry

2 1

Re: [Python-Dev] Dropping __init__.py requirement for subpackages
by Phillip J. Eby 26 Apr '06

26 Apr '06

At 10:16 AM 4/26/2006 -0700, Guido van Rossum wrote: >So I have a very simple proposal: keep the __init__.py requirement for >top-level pacakages, but drop it for subpackages. Note that many tools exist which have grown to rely on the presence of __init__ modules. Also, although your proposal would allow imports to work reasonably well, tools that are actively looking for packages would need to have some way to distinguish package directories from others. My counter-proposal: to be considered a package, a directory must contain at least one module (which of course can be __init__). This allows the "is it a package?" question to be answered with only one directory read, as is the case now. Think of it also as a nudge in favor of "flat is better than nested". This tweak would also make it usable for top-level directories, since the mere presence of a 'time' directory wouldn't get in the way of anything. The thing more likely to have potential for problems is that many Python projects have a "test" directory that isn't intended to be a package, and thus may interfere with imports from the stdlib 'test' package. Whether this is really a problem or not, I don't know. But, we could treat packages without __init__ as namespace packages. That is, set their __path__ to encompass similarly-named directories already on sys.path, so that the init-less package doesn't interfere with other packages that have the same name. This would require a bit of expansion to PEP 302, but probably not much. Most of the rest is existing technology, and we've already begun migrating stdlib modules away from doing their own hunting for __init__ and other files, towards using the pkgutil API. By the way, one small precedent for packages without __init__: setuptools generates such packages using .pth files when a package is split between different distributions but are being installed by a system packaging tool. In such cases, *both* parts of the package can't include an __init__, because the packaging tool (e.g. RPM) is going to complain that the shared file is a conflict. So setuptools generates a .pth file that creates a module object with the right name and initializes its __path__ to point to the __init__-less directory. >This should be a small change. Famous last words. :) There's a bunch of tools that it's not going to work properly with, and not just in today's stdlib. (Think documentation tools, distutils extensions, IDEs...) Are you sure you wouldn't rather just write a GoogleImporter class to fix this problem? Append it to sys.path_hooks, clear sys.path_importer_cache, and you're all set. For that matter, if you have only one top-level package, put the class and the installation code in that top-level __init__, and you're set to go. And that approach will work with Python back to version 2.3; no waiting for an upgrade (unless Google is still using 2.2, of course). Let's see, the code would look something like: class GoogleImporter: def __init__(self, path): if not os.path.isdir(path): raise ImportError("Not for me") self.path = os.path.realpath(path) def find_module(self, fullname, path=None): # Note: we ignore 'path' argument since it is only used via meta_path subname = fullname.split(".")[-1] if os.path.isdir(os.path.join(self.path, subname)): return self path = [self.path] try: file, filename, etc = imp.find_module(subname, path) except ImportError: return None return ImpLoader(fullname, file, filename, etc) def load_module(self, fullname): import sys, new subname = fullname.split(".")[-1] path = os.path.join(self.path, subname) module = sys.modules.setdefault(fullname, new.module(fullname)) module.__dict__.setdefault('__path__',[]).append(path) return module class ImpLoader: def __init__(self, fullname, file, filename, etc): self.file = file self.filename = filename self.fullname = fullname self.etc = etc def load_module(self, fullname): try: mod = imp.load_module(fullname, self.file, self.filename, self.etc) finally: if self.file: self.file.close() return mod import sys sys.path_hooks.append(GoogleImporter) sys.path_importer_cache.clear()

4 6