[Python-Dev] startup time repeated? why not daemon

Brett Cannon brett at python.org
Fri Jul 21 12:52:25 EDT 2017


On Thu, 20 Jul 2017 at 22:11 Chris Jerdonek <chris.jerdonek at gmail.com>
wrote:

> On Thu, Jul 20, 2017 at 8:49 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > ...
> > * Lazy loading can have a significant impact on startup time, as it
> > means you don't have to pay for the cost of finding and loading
> > modules that you don't actually end up using on that particular run
>

It should be mentioned that I have started designing an API to make using
lazy loading much easier in Python 3.7 (i.e. "calling a single function"
easier), but I still have to write the tests and such before I propose a
patch and it will still be mainly for apps that know what they are doing
since lazy loading makes debugging import errors harder.


> >
> > We've historically resisted adopting these techniques for the standard
> > library because they *do* make things more complicated *and* harder to
> > debug relative to plain old eagerly imported dynamic Python code.
> > However, if we're going to recommend them as good practices for 3rd
> > party developers looking to optimise the startup time of their Python
> > applications, then it makes sense for us to embrace them for the
> > standard library as well, rather than having our first reaction be to
> > write more hand-crafted C code.
>
> Are there any good write-ups of best practices and techniques in this
> area for applications (other than obvious things like avoiding
> unnecessary imports)? I'm thinking of things like how to structure
> your project, things to look for, developer tools that might help, and
> perhaps third-party runtime libraries?
>

Nothing beyond "profile your application" and "don't do stuff during import
as a side-effect" that I'm aware of.

-Brett


>
> --Chris
>
>
>
> >
> > On that last point, it's also worth keeping in mind that we have a
> > much harder time finding new C-level contributors than we do new
> > Python-level ones, and have every reason to expect that problem to get
> > worse over time rather than better (since writing and maintaining
> > handcrafted C code is likely to go the way of writing and maintaining
> > handcrafted assembly code as a skillset: while it will still be
> > genuinely necessary in some contexts, it will also be an increasingly
> > niche technical specialty).
> >
> > Starting to migrate to using Cython for our acceleration modules
> > instead of plain C should thus prove to be a win for everyone:
> >
> > - Cython structurally avoids a lot of typical bugs that arise in
> > hand-coded extensions (e.g. refcount bugs)
> > - by design, it's much easier to mentally switch between Python &
> > Cython than it is between Python & C
> > - Cython accelerated modules are easier to adapt to other interpeter
> > implementations than handcrafted C modules
> > - keeping Python modules and their C accelerated counterparts in sync
> > will be easier, as they'll mostly be using the same code
> > - we'd be able to start writing C API test cases in Cython rather than
> > in handcrafted C (which currently mostly translates to only testing
> > them indirectly)
> > - CPython's own test suite would naturally help test Cython
> > compatibility with any C API updates
> > - we'd have an inherent incentive to help enhance Cython to take
> > advantage of new C API features
> >
> > The are some genuine downsides in increasing the complexity of
> > bootstrapping CPython when all you're starting with is a VCS clone and
> > a C compiler, but those complications are ultimately no worse than
> > those we already have with Argument Clinic, and hence amenable to the
> > same solution: if we need to, we can check in the generated C files in
> > order to make bootstrapping easier.
> >
> > Cheers,
> > Nick.
> >
> > --
> > Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170721/cb443c29/attachment.html>


More information about the Python-Dev mailing list