[Python-Dev] startup time repeated? why not daemon

[Chris Jerdonek]

On Sun, Jul 23, 2017 at 5:57 AM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> We already did that. See _bootlocale for example. (Maybe also
> _collecctions_abc?)

I was asking more in the context of recommended practices for
third-party developers, as Nick mentioned earlier, because it's not a
strategy I've ever seen mentioned (and common practice is to group
only by functionality).

It's good to know re: locale and collections though. Incidentally,
from the issue thread it doesn't look like _bootlocale was motivated
primarily by startup time, but _collections_abc was:

locale: http://bugs.python.org/issue9548
collections.abc: http://bugs.python.org/issue19218

--Chris

>
> Victor
>
> Le 22 juil. 2017 07:20, "Chris Jerdonek" <chris.jerdonek at gmail.com> a écrit
> :
>>
>> On Fri, Jul 21, 2017 at 9:52 AM, Brett Cannon <brett at python.org> wrote:
>> > On Thu, 20 Jul 2017 at 22:11 Chris Jerdonek <chris.jerdonek at gmail.com>
>> > wrote:
>> >> On Thu, Jul 20, 2017 at 8:49 PM, Nick Coghlan <ncoghlan at gmail.com>
>> >> wrote:
>> >> > ...
>> >> > * Lazy loading can have a significant impact on startup time, as it
>> >> > means you don't have to pay for the cost of finding and loading
>> >> > modules that you don't actually end up using on that particular run
>> >
>> > It should be mentioned that I have started designing an API to make
>> > using
>> > lazy loading much easier in Python 3.7 (i.e. "calling a single function"
>> > easier), but I still have to write the tests and such before I propose a
>> > patch and it will still be mainly for apps that know what they are doing
>> > since lazy loading makes debugging import errors harder.
>> > ...
>> >> > However, if we're going to recommend them as good practices for 3rd
>> >> > party developers looking to optimise the startup time of their Python
>> >> > applications, then it makes sense for us to embrace them for the
>> >> > standard library as well, rather than having our first reaction be to
>> >> > write more hand-crafted C code.
>> >>
>> >> Are there any good write-ups of best practices and techniques in this
>> >> area for applications (other than obvious things like avoiding
>> >> unnecessary imports)? I'm thinking of things like how to structure
>> >> your project, things to look for, developer tools that might help, and
>> >> perhaps third-party runtime libraries?
>> >
>> > Nothing beyond "profile your application" and "don't do stuff during
>> > import
>> > as a side-effect" that I'm aware of.
>>
>> One "project structure" idea of the sort I had in mind is to move
>> frequently used functions in a module into their own module. This way
>> the most common paths of execution don't load unneeded functions.
>> Following this line of reasoning could lead to grouping functions in
>> an application by when they're needed instead of by what they do,
>> which is different from what we normally see. I don't recall seeing
>> advice like this anywhere, so maybe the trade-offs aren't worth it.
>> Thoughts?
>>
>> --Chris
>>
>>
>> >
>> > -Brett
>> >
>> >>
>> >>
>> >> --Chris
>> >>
>> >>
>> >>
>> >> >
>> >> > On that last point, it's also worth keeping in mind that we have a
>> >> > much harder time finding new C-level contributors than we do new
>> >> > Python-level ones, and have every reason to expect that problem to
>> >> > get
>> >> > worse over time rather than better (since writing and maintaining
>> >> > handcrafted C code is likely to go the way of writing and maintaining
>> >> > handcrafted assembly code as a skillset: while it will still be
>> >> > genuinely necessary in some contexts, it will also be an increasingly
>> >> > niche technical specialty).
>> >> >
>> >> > Starting to migrate to using Cython for our acceleration modules
>> >> > instead of plain C should thus prove to be a win for everyone:
>> >> >
>> >> > - Cython structurally avoids a lot of typical bugs that arise in
>> >> > hand-coded extensions (e.g. refcount bugs)
>> >> > - by design, it's much easier to mentally switch between Python &
>> >> > Cython than it is between Python & C
>> >> > - Cython accelerated modules are easier to adapt to other interpeter
>> >> > implementations than handcrafted C modules
>> >> > - keeping Python modules and their C accelerated counterparts in sync
>> >> > will be easier, as they'll mostly be using the same code
>> >> > - we'd be able to start writing C API test cases in Cython rather
>> >> > than
>> >> > in handcrafted C (which currently mostly translates to only testing
>> >> > them indirectly)
>> >> > - CPython's own test suite would naturally help test Cython
>> >> > compatibility with any C API updates
>> >> > - we'd have an inherent incentive to help enhance Cython to take
>> >> > advantage of new C API features
>> >> >
>> >> > The are some genuine downsides in increasing the complexity of
>> >> > bootstrapping CPython when all you're starting with is a VCS clone
>> >> > and
>> >> > a C compiler, but those complications are ultimately no worse than
>> >> > those we already have with Argument Clinic, and hence amenable to the
>> >> > same solution: if we need to, we can check in the generated C files
>> >> > in
>> >> > order to make bootstrapping easier.
>> >> >
>> >> > Cheers,
>> >> > Nick.
>> >> >
>> >> > --
>> >> > Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>> >> > _______________________________________________
>> >> > Python-Dev mailing list
>> >> > Python-Dev at python.org
>> >> > https://mail.python.org/mailman/listinfo/python-dev
>> >> > Unsubscribe:
>> >> >
>> >> > https://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com
>> >> _______________________________________________
>> >> Python-Dev mailing list
>> >> Python-Dev at python.org
>> >> https://mail.python.org/mailman/listinfo/python-dev
>> >> Unsubscribe:
>> >> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>>
>> On Fri, Jul 21, 2017 at 9:52 AM, Brett Cannon <brett at python.org> wrote:
>> >
>> >
>> > On Thu, 20 Jul 2017 at 22:11 Chris Jerdonek <chris.jerdonek at gmail.com>
>> > wrote:
>> >>
>> >> On Thu, Jul 20, 2017 at 8:49 PM, Nick Coghlan <ncoghlan at gmail.com>
>> >> wrote:
>> >> > ...
>> >> > * Lazy loading can have a significant impact on startup time, as it
>> >> > means you don't have to pay for the cost of finding and loading
>> >> > modules that you don't actually end up using on that particular run
>> >
>> >
>> > It should be mentioned that I have started designing an API to make
>> > using
>> > lazy loading much easier in Python 3.7 (i.e. "calling a single function"
>> > easier), but I still have to write the tests and such before I propose a
>> > patch and it will still be mainly for apps that know what they are doing
>> > since lazy loading makes debugging import errors harder.
>> >
>> >>
>> >> >
>> >> > We've historically resisted adopting these techniques for the
>> >> > standard
>> >> > library because they *do* make things more complicated *and* harder
>> >> > to
>> >> > debug relative to plain old eagerly imported dynamic Python code.
>> >> > However, if we're going to recommend them as good practices for 3rd
>> >> > party developers looking to optimise the startup time of their Python
>> >> > applications, then it makes sense for us to embrace them for the
>> >> > standard library as well, rather than having our first reaction be to
>> >> > write more hand-crafted C code.
>> >>
>> >> Are there any good write-ups of best practices and techniques in this
>> >> area for applications (other than obvious things like avoiding
>> >> unnecessary imports)? I'm thinking of things like how to structure
>> >> your project, things to look for, developer tools that might help, and
>> >> perhaps third-party runtime libraries?
>> >
>> >
>> > Nothing beyond "profile your application" and "don't do stuff during
>> > import
>> > as a side-effect" that I'm aware of.
>> >
>> > -Brett
>> >
>> >>
>> >>
>> >> --Chris
>> >>
>> >>
>> >>
>> >> >
>> >> > On that last point, it's also worth keeping in mind that we have a
>> >> > much harder time finding new C-level contributors than we do new
>> >> > Python-level ones, and have every reason to expect that problem to
>> >> > get
>> >> > worse over time rather than better (since writing and maintaining
>> >> > handcrafted C code is likely to go the way of writing and maintaining
>> >> > handcrafted assembly code as a skillset: while it will still be
>> >> > genuinely necessary in some contexts, it will also be an increasingly
>> >> > niche technical specialty).
>> >> >
>> >> > Starting to migrate to using Cython for our acceleration modules
>> >> > instead of plain C should thus prove to be a win for everyone:
>> >> >
>> >> > - Cython structurally avoids a lot of typical bugs that arise in
>> >> > hand-coded extensions (e.g. refcount bugs)
>> >> > - by design, it's much easier to mentally switch between Python &
>> >> > Cython than it is between Python & C
>> >> > - Cython accelerated modules are easier to adapt to other interpeter
>> >> > implementations than handcrafted C modules
>> >> > - keeping Python modules and their C accelerated counterparts in sync
>> >> > will be easier, as they'll mostly be using the same code
>> >> > - we'd be able to start writing C API test cases in Cython rather
>> >> > than
>> >> > in handcrafted C (which currently mostly translates to only testing
>> >> > them indirectly)
>> >> > - CPython's own test suite would naturally help test Cython
>> >> > compatibility with any C API updates
>> >> > - we'd have an inherent incentive to help enhance Cython to take
>> >> > advantage of new C API features
>> >> >
>> >> > The are some genuine downsides in increasing the complexity of
>> >> > bootstrapping CPython when all you're starting with is a VCS clone
>> >> > and
>> >> > a C compiler, but those complications are ultimately no worse than
>> >> > those we already have with Argument Clinic, and hence amenable to the
>> >> > same solution: if we need to, we can check in the generated C files
>> >> > in
>> >> > order to make bootstrapping easier.
>> >> >
>> >> > Cheers,
>> >> > Nick.
>> >> >
>> >> > --
>> >> > Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>> >> > _______________________________________________
>> >> > Python-Dev mailing list
>> >> > Python-Dev at python.org
>> >> > https://mail.python.org/mailman/listinfo/python-dev
>> >> > Unsubscribe:
>> >> >
>> >> > https://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com
>> >> _______________________________________________
>> >> Python-Dev mailing list
>> >> Python-Dev at python.org
>> >> https://mail.python.org/mailman/listinfo/python-dev
>> >> Unsubscribe:
>> >> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com