[Python-ideas] Defer Statement

[Lucas Wiman]

I agree that the stated use cases are better handled with ExitStack.  One
area where `defer` might be useful is in lazy-evaluating global constants.
For example in a genomics library used at my work, one module involves
compiling a large number of regular expressions, and setting them as global
contants in the module, like:

FOO1_re = re.compile(r'...')
FOO_TO_BAR_re = {foo: complicated_computation_of_regex(foo) for foo in
LONG_LIST_OF_THINGS}
...

This utility module is imported in a lot of places in the codebase, which
meant that importing almost anything from our codebase involved
precompiling all these regular expressions, which took around 500ms to to
run the anything (the test runner, manually testing code in the shell,
etc.) It would be ideal to only do these computations if/when they are
needed. This is a more general issue than this specific example, e.g. for
libraries which parse large data sources like pycountry (see my PR
<https://bitbucket.org/flyingcircus/pycountry/pull-requests/10/improve-module-loading-time-by-lazy/diff>
for one possible ugly solution using proxy objects; the author instead went
with the simpler, less general solution of manually deciding when the data
is needed). See also django.utils.functional.lazy
<https://docs.djangoproject.com/en/1.11/_modules/django/utils/functional/>,
which is used extensively in the framework.

A statement like: `defer FOO = lengthy_computation_of_foo()` which deferred
the lengthy computation until it is used for something would be useful to
allow easily fixing these issues without writing ugly hacks like proxy
objects or refactoring code into high-overhead cached properties or the
like.

Alternatively, if there are less ugly or bespoke ways to handle this kind
of issue, I'd be interested in hearing them.

Best,
Lucas



On Sat, Jun 3, 2017 at 11:38 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 3 June 2017 at 22:24, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > So while I'm definitely sympathetic to the use case (otherwise
> > ExitStack wouldn't have a callback() method), "this would be useful"
> > isn't a sufficient argument in this particular case - what's needed is
> > a justification that this pattern of resource management is common
> > enough to justify giving functions an optional implicit ExitStack
> > instance and assigning a dedicated keyword for adding entries to it.
>
> It occurred to me that I should elaborate a bit further here, and
> point out explicitly that one of the main benefits of ExitStack (and,
> indeed, the main reason it exists) is that it allows resource
> lifecycle management to be deterministic, *without* necessarily tying
> it to function calls or with statements.
>
> The behave BDD test framework, for example, defines hooks that run
> before and after each feature and scenario, as well as before and
> after the entire test run. I use those to set up "scenario_cleanup",
> "_feature_cleanup" and "_global_cleanup" stacks as part of the testing
> context: https://github.com/leapp-to/prototype/blob/master/
> integration-tests/features/environment.py#L49
>
> If a test step implementor allocates a resource that needs to be
> cleaned up, they register it with "context.scenario_cleanup", and then
> the "after scenario" hook takes care of closing the ExitStack instance
> and cleaning everything up appropriately.
>
> For me, that kind of situation is when I'm most inclined to reach for
> ExitStack, whereas when the cleanup needs align with the function call
> stack, I'm more likely to reach for contexlib.contextmanager or an
> explicit try/finally.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20170604/1fc8b7a9/attachment-0001.html>