[Python-Dev] startup time repeated? why not daemon

Victor Stinner victor.stinner at gmail.com
Sun Jul 23 08:57:59 EDT 2017


We already did that. See _bootlocale for example. (Maybe also
_collecctions_abc?)

Victor

Le 22 juil. 2017 07:20, "Chris Jerdonek" <chris.jerdonek at gmail.com> a
écrit :

> On Fri, Jul 21, 2017 at 9:52 AM, Brett Cannon <brett at python.org> wrote:
> > On Thu, 20 Jul 2017 at 22:11 Chris Jerdonek <chris.jerdonek at gmail.com>
> > wrote:
> >> On Thu, Jul 20, 2017 at 8:49 PM, Nick Coghlan <ncoghlan at gmail.com>
> wrote:
> >> > ...
> >> > * Lazy loading can have a significant impact on startup time, as it
> >> > means you don't have to pay for the cost of finding and loading
> >> > modules that you don't actually end up using on that particular run
> >
> > It should be mentioned that I have started designing an API to make using
> > lazy loading much easier in Python 3.7 (i.e. "calling a single function"
> > easier), but I still have to write the tests and such before I propose a
> > patch and it will still be mainly for apps that know what they are doing
> > since lazy loading makes debugging import errors harder.
> > ...
> >> > However, if we're going to recommend them as good practices for 3rd
> >> > party developers looking to optimise the startup time of their Python
> >> > applications, then it makes sense for us to embrace them for the
> >> > standard library as well, rather than having our first reaction be to
> >> > write more hand-crafted C code.
> >>
> >> Are there any good write-ups of best practices and techniques in this
> >> area for applications (other than obvious things like avoiding
> >> unnecessary imports)? I'm thinking of things like how to structure
> >> your project, things to look for, developer tools that might help, and
> >> perhaps third-party runtime libraries?
> >
> > Nothing beyond "profile your application" and "don't do stuff during
> import
> > as a side-effect" that I'm aware of.
>
> One "project structure" idea of the sort I had in mind is to move
> frequently used functions in a module into their own module. This way
> the most common paths of execution don't load unneeded functions.
> Following this line of reasoning could lead to grouping functions in
> an application by when they're needed instead of by what they do,
> which is different from what we normally see. I don't recall seeing
> advice like this anywhere, so maybe the trade-offs aren't worth it.
> Thoughts?
>
> --Chris
>
>
> >
> > -Brett
> >
> >>
> >>
> >> --Chris
> >>
> >>
> >>
> >> >
> >> > On that last point, it's also worth keeping in mind that we have a
> >> > much harder time finding new C-level contributors than we do new
> >> > Python-level ones, and have every reason to expect that problem to get
> >> > worse over time rather than better (since writing and maintaining
> >> > handcrafted C code is likely to go the way of writing and maintaining
> >> > handcrafted assembly code as a skillset: while it will still be
> >> > genuinely necessary in some contexts, it will also be an increasingly
> >> > niche technical specialty).
> >> >
> >> > Starting to migrate to using Cython for our acceleration modules
> >> > instead of plain C should thus prove to be a win for everyone:
> >> >
> >> > - Cython structurally avoids a lot of typical bugs that arise in
> >> > hand-coded extensions (e.g. refcount bugs)
> >> > - by design, it's much easier to mentally switch between Python &
> >> > Cython than it is between Python & C
> >> > - Cython accelerated modules are easier to adapt to other interpeter
> >> > implementations than handcrafted C modules
> >> > - keeping Python modules and their C accelerated counterparts in sync
> >> > will be easier, as they'll mostly be using the same code
> >> > - we'd be able to start writing C API test cases in Cython rather than
> >> > in handcrafted C (which currently mostly translates to only testing
> >> > them indirectly)
> >> > - CPython's own test suite would naturally help test Cython
> >> > compatibility with any C API updates
> >> > - we'd have an inherent incentive to help enhance Cython to take
> >> > advantage of new C API features
> >> >
> >> > The are some genuine downsides in increasing the complexity of
> >> > bootstrapping CPython when all you're starting with is a VCS clone and
> >> > a C compiler, but those complications are ultimately no worse than
> >> > those we already have with Argument Clinic, and hence amenable to the
> >> > same solution: if we need to, we can check in the generated C files in
> >> > order to make bootstrapping easier.
> >> >
> >> > Cheers,
> >> > Nick.
> >> >
> >> > --
> >> > Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> >> > _______________________________________________
> >> > Python-Dev mailing list
> >> > Python-Dev at python.org
> >> > https://mail.python.org/mailman/listinfo/python-dev
> >> > Unsubscribe:
> >> > https://mail.python.org/mailman/options/python-dev/
> chris.jerdonek%40gmail.com
> >> _______________________________________________
> >> Python-Dev mailing list
> >> Python-Dev at python.org
> >> https://mail.python.org/mailman/listinfo/python-dev
> >> Unsubscribe:
> >> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
> On Fri, Jul 21, 2017 at 9:52 AM, Brett Cannon <brett at python.org> wrote:
> >
> >
> > On Thu, 20 Jul 2017 at 22:11 Chris Jerdonek <chris.jerdonek at gmail.com>
> > wrote:
> >>
> >> On Thu, Jul 20, 2017 at 8:49 PM, Nick Coghlan <ncoghlan at gmail.com>
> wrote:
> >> > ...
> >> > * Lazy loading can have a significant impact on startup time, as it
> >> > means you don't have to pay for the cost of finding and loading
> >> > modules that you don't actually end up using on that particular run
> >
> >
> > It should be mentioned that I have started designing an API to make using
> > lazy loading much easier in Python 3.7 (i.e. "calling a single function"
> > easier), but I still have to write the tests and such before I propose a
> > patch and it will still be mainly for apps that know what they are doing
> > since lazy loading makes debugging import errors harder.
> >
> >>
> >> >
> >> > We've historically resisted adopting these techniques for the standard
> >> > library because they *do* make things more complicated *and* harder to
> >> > debug relative to plain old eagerly imported dynamic Python code.
> >> > However, if we're going to recommend them as good practices for 3rd
> >> > party developers looking to optimise the startup time of their Python
> >> > applications, then it makes sense for us to embrace them for the
> >> > standard library as well, rather than having our first reaction be to
> >> > write more hand-crafted C code.
> >>
> >> Are there any good write-ups of best practices and techniques in this
> >> area for applications (other than obvious things like avoiding
> >> unnecessary imports)? I'm thinking of things like how to structure
> >> your project, things to look for, developer tools that might help, and
> >> perhaps third-party runtime libraries?
> >
> >
> > Nothing beyond "profile your application" and "don't do stuff during
> import
> > as a side-effect" that I'm aware of.
> >
> > -Brett
> >
> >>
> >>
> >> --Chris
> >>
> >>
> >>
> >> >
> >> > On that last point, it's also worth keeping in mind that we have a
> >> > much harder time finding new C-level contributors than we do new
> >> > Python-level ones, and have every reason to expect that problem to get
> >> > worse over time rather than better (since writing and maintaining
> >> > handcrafted C code is likely to go the way of writing and maintaining
> >> > handcrafted assembly code as a skillset: while it will still be
> >> > genuinely necessary in some contexts, it will also be an increasingly
> >> > niche technical specialty).
> >> >
> >> > Starting to migrate to using Cython for our acceleration modules
> >> > instead of plain C should thus prove to be a win for everyone:
> >> >
> >> > - Cython structurally avoids a lot of typical bugs that arise in
> >> > hand-coded extensions (e.g. refcount bugs)
> >> > - by design, it's much easier to mentally switch between Python &
> >> > Cython than it is between Python & C
> >> > - Cython accelerated modules are easier to adapt to other interpeter
> >> > implementations than handcrafted C modules
> >> > - keeping Python modules and their C accelerated counterparts in sync
> >> > will be easier, as they'll mostly be using the same code
> >> > - we'd be able to start writing C API test cases in Cython rather than
> >> > in handcrafted C (which currently mostly translates to only testing
> >> > them indirectly)
> >> > - CPython's own test suite would naturally help test Cython
> >> > compatibility with any C API updates
> >> > - we'd have an inherent incentive to help enhance Cython to take
> >> > advantage of new C API features
> >> >
> >> > The are some genuine downsides in increasing the complexity of
> >> > bootstrapping CPython when all you're starting with is a VCS clone and
> >> > a C compiler, but those complications are ultimately no worse than
> >> > those we already have with Argument Clinic, and hence amenable to the
> >> > same solution: if we need to, we can check in the generated C files in
> >> > order to make bootstrapping easier.
> >> >
> >> > Cheers,
> >> > Nick.
> >> >
> >> > --
> >> > Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> >> > _______________________________________________
> >> > Python-Dev mailing list
> >> > Python-Dev at python.org
> >> > https://mail.python.org/mailman/listinfo/python-dev
> >> > Unsubscribe:
> >> > https://mail.python.org/mailman/options/python-dev/
> chris.jerdonek%40gmail.com
> >> _______________________________________________
> >> Python-Dev mailing list
> >> Python-Dev at python.org
> >> https://mail.python.org/mailman/listinfo/python-dev
> >> Unsubscribe:
> >> https://mail.python.org/mailman/options/python-dev/brett%40python.org
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> victor.stinner%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170723/2693aebf/attachment.html>


More information about the Python-Dev mailing list