Sorry for top posting! I'm on a phone. I still think the better way to solve the custom dir() would be to change the module __dir__ method to check if __all__ is defined and use it to generate the result if it exists. This seems like a logical enhancement to me, and I'm planning on writing a patch to implement this. Whether it would be accepted is still an open issue though. Cody On Sep 10, 2017 3:19 PM, "Nathaniel Smith" wrote: The main two use cases I know of for this and PEP 549 are lazy imports of submodules, and deprecating attributes. If we assume that you only want lazy imports to show up in dir() and don't want deprecated attributes to show up in dir() (and I'm not sure this is what you want 100% of the time, but it seems like the most reasonable default to me), then currently you need one of the PEPs for one of the cases and the other PEP for the other case. Would it make more sense to add direct support for lazy imports and attribute deprecation to ModuleType? This might look something like metamodule's FancyModule type: https://github.com/njsmith/metamodule/blob/ee54d49100a9a06 ffff341bb10a4d3549642139f/metamodule.py#L20 -n

I don't think submodules are automatically imported, unless there are import statements in __init__.py. On Sun, Sep 10, 2017 at 8:08 PM, INADA Naoki wrote:

It looks simple and easy to understand.

To achieve lazy import without breaking backward compatibility, I want to add one more rule: When package defines both of __getattr__ and __all__, automatic import of submodules are disabled (sorry, I don't have pointer to specification about this behavior).

For example, some modules depends on email.parser or email.feedparser. But since email/__init__.py uses __all__, all submodules are imported eagerly.

See https://github.com/python/cpython/blob/master/Lib/email/ __init__.py#L7-L25

Changing __all__ will break backward compatibility. With __getattr__, this can be lazy import:

import importlib

def __getattr__(name): if name in __all__: return importlib.import_module("." + name, __name__) raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

Regards,

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido)

There's no need for shame! I regularly find out that there are Python features I didn't know about. It's called perpetual learning. :-) On Sun, Sep 10, 2017 at 9:02 PM, INADA Naoki wrote:

Oh, I'm shame myself.

Only when `from email import *` is used, __all__ submodules are imported. INADA Naoki

On Mon, Sep 11, 2017 at 12:17 PM, Guido van Rossum wrote:

...

I don't think submodules are automatically imported, unless there are import statements in __init__.py.

On Sun, Sep 10, 2017 at 8:08 PM, INADA Naoki wrote:

...

It looks simple and easy to understand.

To achieve lazy import without breaking backward compatibility, I want to add one more rule: When package defines both of __getattr__

and

...

__all__, automatic import of submodules are disabled (sorry, I don't have pointer to specification about this behavior).

For example, some modules depends on email.parser or email.feedparser. But since email/__init__.py uses __all__, all submodules are imported eagerly.

See https://github.com/python/cpython/blob/master/Lib/email/ __init__.py#L7-L25

Changing __all__ will break backward compatibility. With __getattr__, this can be lazy import:

import importlib

def __getattr__(name): if name in __all__: return importlib.import_module("." + name, __name__) raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

Regards,

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido)

-- --Guido van Rossum (python.org/~guido)

If I recall there was a proposal a few months for a "lazy" keyword that would render anything lazy, including imports. Instead of just adding laziness on generators, the on imports, then who knows where, maybe it's time to consider laziness is a hell of a good general concept and try to generalize it ? For imports, that would mean: lazy from module import stuff lazy import foo For the rest, bar = lazy 1 + 1 When you think about it, it's syntaxic sugar to avoid manually wrapping everything in functions, storying stuff in closure and calling that later. Le 12/09/2017 à 10:26, Ivan Levkivskyi a écrit :

@Anthony

...

module.__getattr__ works pretty well for normal access, after being imported by another module, but it doesn't properly trigger loading by functions defined in the module's own namespace.

The idea of my PEP is to be very simple (both semantically and in terms of implementation). This is why I don't want to add any complex logic. People who will want to use __getattr__ for lazy loading still can do this by importing submodules.

@Nathaniel @INADA

...

The main two use cases I know of for this and PEP 549 are lazy imports of submodules, and deprecating attributes.

Yes, lazy loading seems to be a popular idea :-) I will add the simple recipe by Inada to the PEP since it will already work.

@Cody

...

I still think the better way to solve the custom dir() would be to change the module __dir__ method to check if __all__ is defined and use it to generate the result if it exists. This seems like a logical enhancement to me, and I'm planning on writing a patch to implement this. Whether it would be accepted is still an open issue though.

This seems a reasonable rule to me, I can also make this patch if you will not have time.

@Guido What do you think about the above idea?

-- Ivan

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On 10 November 2017 at 22:27, Guido van Rossum wrote:

Picking up this thread as part of the PEP 562 and PEP 549 review. I like PEP 562 most, but I propose to add special-casing for `__dir__`. Not quite as proposed above (making the C level module_dir() look for `__all__`) but a bit more general -- making module_dir() look for `__dir__` and call that if present and callable. Ivan what do you think of that idea? It should be simple to add to your existing implementation. (https://github.com/ ilevkivskyi/cpython/pull/3#issuecomment-343591293)

I like this idea. I was thinking about yet another option: *extending* the result of current dir() search by contents __all__ if present (not just returning contents of __all__). But it looks like your idea covers more use cases, so I would stick with your idea. -- Ivan

On Sep 12, 2017 7:08 AM, "Ionel Cristian Mărieș via Python-ideas" < python-ideas@python.org> wrote: Wouldn't a better approach be a way to customize the type of the module? That would allow people to define behavior for almost anything (__call__, __getattr__, __setattr__, __dir__, various operators etc). This question shouldn't exist "why can't I customize behavior X in a module when I can do it for a class". Why go half-way. If you're ok with replacing the object in sys.modules then the ability to totally customize your module's type has existed since the dawn era. And if you're not ok with that, then it's still existed since 3.5 via the mechanism of assigning to __class__ to change the type in-place. So this discussion isn't about adding new functionality per se, but about trying to find some way to provide a little bit of sugar that provides most of the value in a less obscure way. (And unfortunately there's a chicken and egg problem for using custom module types *without* the __class__ assignment hack, because you can't load any code from a package until after you've created the top level module object. So we've kind of taken custom module types as far as they can go already.) -n

On Tue, Sep 12, 2017 at 10:32 PM Nathaniel Smith wrote:

If you're ok with replacing the object in sys.modules then the ability to totally customize your module's type has existed since the dawn era.

I'm down with that. Just make it easier, mucking with sys.modules ain't a walk in a park, and there's the boilerplate and the crazy issues with interpreter shutdown. -- Thanks, -- Ionel Cristian Mărieș, http://blog.ionelmc.ro

On Tue, Nov 14, 2017 at 12:24 AM, Serhiy Storchaka wrote:

10.09.17 21:48, Ivan Levkivskyi пише:

...

# lib.py

from warnings import warn

deprecated_names = ["old_function", ...]

def _deprecated_old_function(arg, other): ...

def __getattr__(name): if name in deprecated_names: warn(f"{name} is deprecated", DeprecationWarning) return globals()[f"_deprecated_{name}"] raise AttributeError(f"module {__name__} has no attribute {name}")

# main.py

from lib import old_function # Works, but emits the warning

I think the PEP should provide better examples, because they will be copied in third-party code.

For keeping all functionality (in particularly be pickleable) the deprecated function should preserve its name.

def old_function(arg, other): ... _deprecated_old_function = old_function del old_function

or

def _deprecated_old_function(arg, other): ... _deprecated_old_function.__name__ = 'old_function' _deprecated_old_function.__qualname__ = 'old_function'

(I prefer the former variant.)

However the latter could be done by a decorator.

I'm wondering if it is worth to provide a special helper that will rename the deprecated function and create the corresponding __getattr__ function.

It could be possible to create helpers that implement module-level properties too.

OTOH not everyone cares about whether their functions are picklable. Maybe a discussion of how to make deprecated functions picklable could be in an Appendix? And another example of how to implement more functional lazy loading. Or even more complete examples could be maintained as a separate GitHub repo (like Ethan did for PEP 561). -- --Guido van Rossum (python.org/~guido)