Mailman 3 April 2006 - Python-Dev

PY_FORMAT_SIZE_T warnings on OS X
by Brett Cannon April 27, 2006

April 27, 2006

I think these are all Tim's fault =) : Objects/object.c: In function '_Py_NegativeRefcount': Objects/object.c:144: warning: format '%d' expects type 'int', but argument 7 has type 'Py_ssize_t' Objects/stringobject.c: In function 'PyString_FromFormatV': Objects/stringobject.c:278: warning: format '%u' expects type 'unsigned int', but argument 3 has type 'size_t' Python/pythonrun.c: In function 'Py_Finalize': Python/pythonrun.c:393: warning: format '%d' expects type 'int', but argument 3 has … [View More]

3 9

need info for externally maintained modules PEP
by Brett Cannon April 27, 2006

April 27, 2006

OK, I am going to write the PEP I proposed a week or so ago, listing all modules and packages within the stdlib that are maintained externally so we have a central place to go for contact info or where to report bugs on issues. This should only apply to modules that want bugs reported outside of the Python tracker and have a separate dev track. People who just use the Python repository as their mainline version can just be left out. For each package I need the name of the module, the name of … [View More]

12 20

Re: [Python-Dev] Dropping __init__.py requirement for subpackages
by Phillip J. Eby April 27, 2006

April 27, 2006

At 11:50 AM 4/26/2006 -0700, Guido van Rossum wrote: >I'm not sure what you mean by "one directory read". You'd have to list >the entire directory, which may require reading more than one block if >the directory is large. You have to do this to find an __init__.py too, don't you? Technically, there's going to be a search for a .pyc or .pyo first, anyway. I'm just saying you can stop as soon as you hit an extension that's in imp.get_suffixes(). > > Are you sure you wouldn'… [View More]t rather just write a GoogleImporter class to fix > > this problem? > >No, because that would require more setup code with a requirement to >properly enable it, etc., etc., more failure modes, etc., etc. I don't understand. I thought you said you had only *one* top-level package. Fix *that* package, by putting the code in its __init__.py. Job done. > > Append it to sys.path_hooks, clear sys.path_importer_cache, > > and you're all set. For that matter, if you have only one top-level > > package, put the class and the installation code in that top-level > > __init__, and you're set to go. > >I wish it were that easy. Well, if there's more than one top-level package, put the code in a module called google_imports and "import google_import" in each top-level package's __init__.py. I'm not sure I understand why a solution that works with released versions of Python that allows you to do exactly what you want, is inferior to a hypothetical solution for an unreleased version of Python that forces everybody else to update their tools. Unless of course the problem you're trying to solve is a political one rather than a technical one, that is. Or perhaps it wasn't clear from my explanation that my proposal will work the way you need it to, or I misunderstand what you're trying to do. Anyway, I'm not opposed to the idea of supporting this in future Pythons, but I definitely think it falls under the "but sometimes never is better than RIGHT now" rule where 2.5 is concerned. :) In particular, I'm worried that you're shrugging off the extent of the collateral damage here, and I'd be happiest if we waited until 3.0 before changing this particular rule -- and if we changed it in favor of namespace packages, which will more closely match naive user expectations. However, the "fix the tools" argument is weak, IMO. Zipfile imports have been a fairly half-assed feature for 2.3 and 2.4 because nobody took the time to make the *rest* of the stdlib work with zip imports. It's not a good idea to change machinery like this without considering at least what's going to have to be fixed in the stdlib. At a minimum, pydoc and distutils have embedded assumptions regarding __init__ modules, and I wouldn't be surprised if ihooks, imputil, and others do as well. If we can't keep the stdlib up to date with changes in the language, how can we expect anybody else to keep their code up to date? Finally, as others have pointed out, requiring __init__ at the top level just means that this isn't going to help anybody but Google. ISTM that in most cases outside Google, Python newbies are more likely to be creating top-level packages *first*, so the implicit __init__ doesn't help them. So, to summarize: 1. It only really helps Google 2. It inconveniences others who have to update their tools in order to support people who end up using it (even if by accident) 3. It's not a small change, unless you leave the rest of the stdlib unreviewed for impacts 4. It could be fixed at Google by adding a very small amount of code to the top of your __init__.py files (although apparently this is prevented for mysterious reasons that can't be shared) What's not to like? ;) [View Less]

3 2

Re: [Python-Dev] Dropping __init__.py requirement for subpackages
by Phillip J. Eby April 27, 2006

April 27, 2006

At 04:57 PM 4/26/2006 -0700, Guido van Rossum wrote: >On 4/26/06, Delaney, Timothy (Tim) <tdelaney(a)avaya.com> wrote: > > Possibly. Perhaps it would be useful to have `is_package(dirname)`, > > `is_rootpackage(dirname)` and `is_subpackage(dirname)` functions > > somewhere (pkgutils?). > >YAGNI. Also note that not all modules or packages are represented by >pathnames -- they could live in zip files, or be accessed via whatever >other magic an import … [View More]

1 0

Re: [Python-Dev] Dropping __init__.py requirement for subpackages
by Phillip J. Eby April 27, 2006

April 27, 2006

At 01:10 AM 4/27/2006 +0200, Thomas Wouters wrote: >On 4/27/06, Guido van Rossum <<mailto:guido@python.org>guido(a)python.org> >wrote: >>I'd worry that it'll cause complaints when the warning is incorrect >>and a certain directory is being skipped intentionally. E.g. the >>"string" directory that someone had. Getting a warning like this can >>be just as upsetting to newbies! > >I don't think getting a spurious warning is as upsetting as getting … [View More]

1 0

Re: [Python-Dev] Dropping __init__.py requirement for subpackages
by Delaney, Timothy (Tim) April 27, 2006

April 27, 2006

Guido van Rossum wrote: > The difference is that if you find a valid module package later on the > path, you'll still get warnings. This is the bit I don't like about it. Currently the warnings are displayed as the packages are found. I'd be quite happy with the warnings if they were suppressed until there is an actual ImportError - at which time they'll be displayed. > But occasionally those will be useful > too -- when you are trying to create a package that happens to have … [View More]

2 1

Re: [Python-Dev] Dropping __init__.py requirement for subpackages
by Delaney, Timothy (Tim) April 27, 2006

April 27, 2006

Guido van Rossum wrote: >> http://python.org/sf/1477281 >> >> (You can call it 'oldtimer-repellant' if you want to use it to >> convince people there isn't any *real* backward-compatibility issue.) > > I'd worry that it'll cause complaints when the warning is incorrect > and a certain directory is being skipped intentionally. E.g. the > "string" directory that someone had. Getting a warning like this can > be just as upsetting to newbies! I really … [View More]

2 1

Re: [Python-Dev] Dropping __init__.py requirement for subpackages
by Phillip J. Eby April 27, 2006

April 27, 2006

At 01:49 PM 4/26/2006 -0700, Guido van Rossum wrote: >OK, forget it. I'll face the pitchforks. > >I'm disappointed though -- it sounds like we can never change anything >about Python any more because it will upset the oldtimers. I know exactly how you feel. :) But there's always Python 3.0, and if we're refactoring the import machinery there, we can do this the right way, not just the "right now" way. ;) IMO, if Py3K does this, it can and should be inclusive of top-level … [View More]

2 1

Re: [Python-Dev] Dropping __init__.py requirement for subpackages
by Terry Jones April 27, 2006

April 27, 2006

It might be helpful to consider how people would tackle Guido's problem by pretending that a regular Joe (i.e., someone who couldn't consider changing the semantics of Python itself) had asked this question. I would suggest adding a hook to their version control system to automatically create (and preferably also check out) an __init__.py file whenever a new (source code) directory was placed under version control (supposing you can distinguish source code directories from the check in dirname)… [View More]

2 1

Re: [Python-Dev] Dropping __init__.py requirement for subpackages
by Phillip J. Eby April 27, 2006

April 27, 2006

At 10:16 AM 4/26/2006 -0700, Guido van Rossum wrote: >So I have a very simple proposal: keep the __init__.py requirement for >top-level pacakages, but drop it for subpackages. Note that many tools exist which have grown to rely on the presence of __init__ modules. Also, although your proposal would allow imports to work reasonably well, tools that are actively looking for packages would need to have some way to distinguish package directories from others. My counter-proposal: to be … [View More]considered a package, a directory must contain at least one module (which of course can be __init__). This allows the "is it a package?" question to be answered with only one directory read, as is the case now. Think of it also as a nudge in favor of "flat is better than nested". This tweak would also make it usable for top-level directories, since the mere presence of a 'time' directory wouldn't get in the way of anything. The thing more likely to have potential for problems is that many Python projects have a "test" directory that isn't intended to be a package, and thus may interfere with imports from the stdlib 'test' package. Whether this is really a problem or not, I don't know. But, we could treat packages without __init__ as namespace packages. That is, set their __path__ to encompass similarly-named directories already on sys.path, so that the init-less package doesn't interfere with other packages that have the same name. This would require a bit of expansion to PEP 302, but probably not much. Most of the rest is existing technology, and we've already begun migrating stdlib modules away from doing their own hunting for __init__ and other files, towards using the pkgutil API. By the way, one small precedent for packages without __init__: setuptools generates such packages using .pth files when a package is split between different distributions but are being installed by a system packaging tool. In such cases, *both* parts of the package can't include an __init__, because the packaging tool (e.g. RPM) is going to complain that the shared file is a conflict. So setuptools generates a .pth file that creates a module object with the right name and initializes its __path__ to point to the __init__-less directory. >This should be a small change. Famous last words. :) There's a bunch of tools that it's not going to work properly with, and not just in today's stdlib. (Think documentation tools, distutils extensions, IDEs...) Are you sure you wouldn't rather just write a GoogleImporter class to fix this problem? Append it to sys.path_hooks, clear sys.path_importer_cache, and you're all set. For that matter, if you have only one top-level package, put the class and the installation code in that top-level __init__, and you're set to go. And that approach will work with Python back to version 2.3; no waiting for an upgrade (unless Google is still using 2.2, of course). Let's see, the code would look something like: class GoogleImporter: def __init__(self, path): if not os.path.isdir(path): raise ImportError("Not for me") self.path = os.path.realpath(path) def find_module(self, fullname, path=None): # Note: we ignore 'path' argument since it is only used via meta_path subname = fullname.split(".")[-1] if os.path.isdir(os.path.join(self.path, subname)): return self path = [self.path] try: file, filename, etc = imp.find_module(subname, path) except ImportError: return None return ImpLoader(fullname, file, filename, etc) def load_module(self, fullname): import sys, new subname = fullname.split(".")[-1] path = os.path.join(self.path, subname) module = sys.modules.setdefault(fullname, new.module(fullname)) module.__dict__.setdefault('__path__',[]).append(path) return module class ImpLoader: def __init__(self, fullname, file, filename, etc): self.file = file self.filename = filename self.fullname = fullname self.etc = etc def load_module(self, fullname): try: mod = imp.load_module(fullname, self.file, self.filename, self.etc) finally: if self.file: self.file.close() return mod import sys sys.path_hooks.append(GoogleImporter) sys.path_importer_cache.clear() [View Less]

4 6