It doesn't currently end up on disk. Some tables are partially or completely stored on disk as Python source code (some are partially generated from simple rules), but others are generated by inverting those. That process takes time that could be avoided by storing the generated tables, and storing all of it in a format that doesn't require parsing, compiling and executing (such as a native array). Potentially it could be a win all around if we stopped including the (larger) source files, but that doesn't seem like a good idea for maintaining portability to other implementations. The main thought is making the compiler binary bigger to avoid generating encoding tables at startup. Top-posted from my Windows Phone -----Original Message----- From: "francismb" Sent: ‎1/‎29/‎2016 13:56 To: "python-dev@python.org" Subject: Re: [Python-Dev] More optimisation ideas Hi,

Storing these in static data is a tradeoff between disk space and startup performance, and one I think it likely to be worthwhile.

it's really an important trade off? As far a I understand from your email those modules are always being loaded and the final data created. won't the space be there (on mem or disk)? Thanks in advance! francis _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org

On 30.01.16 18:31, Steve Dower wrote:

On 30Jan2016 0645, Serhiy Storchaka wrote:

...

$ ./python -m timeit -s "import codecs; from encodings.cp437 import decoding_table" -- "codecs.charmap_build(decoding_table)" 100000 loops, best of 3: 4.36 usec per loop

Getting rid from charmap_build() would save you at most 4.4 microseconds per encoding. 0.0005 seconds if you have imported *all* standard encodings!

Just as happy to be proven wrong. Perhaps I misinterpreted my original profiling and then, embarrassingly, ran with the result for a long time without retesting.

AFAIK the most time is spent in system calls like stat or open. Archiving the stdlib into the ZIP file and using zipimport can decrease Python startup time (perhaps there is an open issue about this).

Brett tried freezing the entire stdlib at one point (as we do for parts of importlib) and reported no significant improvement. Since that rules out code compilation as well as the OS calls, it'd seem the priority is to execute less code on startup. Details of that work were posted to python-dev about twelve months ago, IIRC. Maybe a little longer. Top-posted from my Windows Phone -----Original Message----- From: "Serhiy Storchaka" Sent: ‎1/‎30/‎2016 10:22 To: "python-dev@python.org" Subject: Re: [Python-Dev] More optimisation ideas On 30.01.16 18:31, Steve Dower wrote:

On 30Jan2016 0645, Serhiy Storchaka wrote:

...

$ ./python -m timeit -s "import codecs; from encodings.cp437 import decoding_table" -- "codecs.charmap_build(decoding_table)" 100000 loops, best of 3: 4.36 usec per loop

Getting rid from charmap_build() would save you at most 4.4 microseconds per encoding. 0.0005 seconds if you have imported *all* standard encodings!

Just as happy to be proven wrong. Perhaps I misinterpreted my original profiling and then, embarrassingly, ran with the result for a long time without retesting.

AFAIK the most time is spent in system calls like stat or open. Archiving the stdlib into the ZIP file and using zipimport can decrease Python startup time (perhaps there is an open issue about this). _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org

So freezing the stdlib helps on UNIX and not on OS X (if my old testing is still accurate). I guess the next question is what it does on Windows and if we would want to ever consider freezing the stdlib as part of the build process (and if we would want to change the order of importers on sys.meta_path so frozen modules came after file-based ones). On Sun, 31 Jan 2016, 10:43 M.-A. Lemburg wrote:

On 30.01.2016 20:15, Steve Dower wrote:

...

Brett tried freezing the entire stdlib at one point (as we do for parts of importlib) and reported no significant improvement. Since that rules out code compilation as well as the OS calls, it'd seem the priority is to execute less code on startup.

Details of that work were posted to python-dev about twelve months ago, IIRC. Maybe a little longer.

Freezing the entire stdlib does improve the startup time, simply because it removes stat calls, which dominate the startup time at least on Unix.

It also allows sharing the stdlib byte code in memory, since it gets stored in static C structs which the OS will happily mmap into multiple processes for you without any additional effort.

Our eGenix PyRun does exactly that. Even though the original motivation is a different one, the gained improvement in startup time is a nice side effect:

http://www.egenix.com/products/python/PyRun/

Aside: The encodings don't really make much difference here. The dictionaries aren't all that big, so generating them on the fly doesn't really create much overhead. The trade off in terms of maintainability/speed definitely leans toward maintainability. For the larger encoding tables we already have C implementations with appropriate data structures to make lookup speed vs. storage needs efficient.

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Experts (#1, Jan 31 2016)

...

...

...

Python Projects, Coaching and Consulting ... http://www.egenix.com/ Python Database Interfaces ... http://products.egenix.com/ Plone/Zope Database Interfaces ... http://zope.egenix.com/

---

::: We implement business ideas - efficiently in both time and costs :::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/

...

Top-posted from my Windows Phone

-----Original Message----- From: "Serhiy Storchaka" Sent: ‎1/‎30/‎2016 10:22 To: "python-dev@python.org" Subject: Re: [Python-Dev] More optimisation ideas

On 30.01.16 18:31, Steve Dower wrote:

...

On 30Jan2016 0645, Serhiy Storchaka wrote:

...

$ ./python -m timeit -s "import codecs; from encodings.cp437 import decoding_table" -- "codecs.charmap_build(decoding_table)" 100000 loops, best of 3: 4.36 usec per loop

Getting rid from charmap_build() would save you at most 4.4 microseconds per encoding. 0.0005 seconds if you have imported *all* standard encodings!

Just as happy to be proven wrong. Perhaps I misinterpreted my original profiling and then, embarrassingly, ran with the result for a long time without retesting.

AFAIK the most time is spent in system calls like stat or open. Archiving the stdlib into the ZIP file and using zipimport can decrease Python startup time (perhaps there is an open issue about this).

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/mal%40egenix.com

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org

On Mon, 01 Feb 2016 14:12:27 +1100, Steven D'Aprano wrote:

On Sun, Jan 31, 2016 at 08:23:00PM +0000, Brett Cannon wrote:

...

So freezing the stdlib helps on UNIX and not on OS X (if my old testing is still accurate). I guess the next question is what it does on Windows and if we would want to ever consider freezing the stdlib as part of the build process (and if we would want to change the order of importers on sys.meta_path so frozen modules came after file-based ones).

I find that being able to easily open stdlib .py files in a text editor to read the source is extremely valuable. I've learned much more from reading the source than from (e.g.) StackOverflow. Likewise, it's often handy to do a grep over the stdlib. When you talk about freezing the stdlib, what exactly does that mean?

- will the source files still be there?

Well, Brett said it would be optional, though perhaps the above paragraph is asking about doing it in our Windows build. But the linux distros might make also use the option if it exists, so the question is very meaningful. However, you'd have to ask the distro if the source would be shipped in the linux case, and I'd guess not in most cases. I don't know about anyone else, but on my own development systems it is not that unusual for me to *edit* the stdlib files (to add debug prints) while debugging my own programs. Freeze would definitely interfere with that. I could, of course, install a separate source build on my dev system, but I thought it worth mentioning as a factor. On the other hand, if the distros go the way Nick has (I think) been advocating, and have a separate 'system python for system scripts' that is independent of the one installed for user use, having the system-only python be frozen and sourceless would actually make sense on a couple of levels. --David

...

...

...

...

" " == Barry Warsaw writes:

>> On Feb 01, 2016, at 11:40 AM, R. David Murray wrote: >> I don't know about anyone else, but on my own development >> systems it is not that unusual for me to *edit* the stdlib >> files (to add debug prints) while debugging my own programs. >> Freeze would definitely interfere with that. I could, of >> course, install a separate source build on my dev system, but I >> thought it worth mentioning as a factor. [snip] > But even with system scripts, I do need to step through them > occasionally. If it were a matter of changing a shebang or > invoking the script with a different Python > (e.g. /usr/bin/python3s vs. /usr/bin/python3) to get the full > unpacked source, that would be fine. If the stdlib were to use implicit namespace packages ( https://www.python.org/dev/peps/pep-0420/ ) and the various loaders/importers as well, then python could do what I've done with an embedded python application for years. Freeze the stdlib (or put it in a zipfile or whatever is fast). Then arrange PYTHONPATH to first look on the filesystem and then look in the frozen/ziped storage. Normally the filesystem part is empty. So, modules are loaded from the frozen/zip area. But if you wanna override one of the frozen modules simply copy one or more .py files onto the file system. I've been doing this only with modules in the global scope. But implicit namespace packages seem to open the door for this with packages. Mike

On Feb 1, 2016, at 09:59, mike.romberg@comcast.net wrote:

If the stdlib were to use implicit namespace packages ( https://www.python.org/dev/peps/pep-0420/ ) and the various loaders/importers as well, then python could do what I've done with an embedded python application for years. Freeze the stdlib (or put it in a zipfile or whatever is fast). Then arrange PYTHONPATH to first look on the filesystem and then look in the frozen/ziped storage.

This is a great solution for experienced developers, but I think it would be pretty bad for novices or transplants from other languages (maybe even including Python 2). There are already multiple duplicate questions every month on StackOverflow from people asking "how do I find the source to stdlib module X". The canonical answer starts off by explaining how to import the module and use its __file__, which everyone is able to handle. If we have to instead explain how to work out the .py name from the qualified module name, how to work out the stdlib path from sys.path, and then how to find the source from those two things, with the caveat that it may not be installed at all on some platforms, and how to make sure what they're asking about really is a stdlib module, and how to make sure they aren't shadowing it with a module elsewhere on sys.path, that's a lot more complicated. Especially when you consider that some people on Windows and Mac are writing Python scripts without ever learning how to use the terminal or find their Python packages via Explorer/Finder. And meanwhile, other people would be asking why their app runs slower on one machine than another, because they didn't expect that installing python-dev on top of python would slow down startup. Finally, on Linux and Mac, the stdlib will usually be somewhere that's not user-writable--and we shouldn't expect users to have to mess with stuff in /usr/lib or /System/Library even if they do have sudo access. Of course we could put a "stdlib shadow" location on the sys.path and configure it for /usr/local/lib and /Library and/or for somewhere in -, but that just makes the lookup proceed even more complicated--not to mention that we've just added three stat calls to remove one open, at which point the optimization has probably become a pessimization.

On Feb 1, 2016, at 19:44, Terry Reedy wrote:

...

On 2/1/2016 3:39 PM, Andrew Barnert via Python-Dev wrote:

There are already multiple duplicate questions every month on StackOverflow from people asking "how do I find the source to stdlib module X". The canonical answer starts off by explaining how to import the module and use its __file__, which everyone is able to handle.

Perhaps even easier: start IDLE, hit Alt-M, type in module name as one would import it, click OK. If Python source is available, IDLE will open in an editor window. with the path on the title bar.

...

If we have to instead explain how to work out the .py name from the qualified module name, how to work out the stdlib path from sys.path, and then how to find the source from those two things, with the caveat that it may not be installed at all on some platforms, and how to make sure what they're asking about really is a stdlib module, and how to make sure they aren't shadowing it with a module elsewhere on sys.path, that's a lot more complicated.

The windows has the path on the title bar, so one can tell what was loaded.

The point of this thread is the suggestion that the stdlib modules be frozen or stored in a zipfile, unless a user modifies things in some way to make the source accessible. So, if a user hasn't done that (which no novice will know how to do), there won't be a path to show in the title bar, so IDLE won't be any more help than the command line. (I suppose IDLE could grow a new feature to look up "associated source files" for a zipped stdlib or something, but that seems like a pretty big new feature.)

IDLE currently uses imp.find_module (this could be updated), with a backup of __import__(...).__file__, so it will load non-stdlib files that can be imported.

...

Finally, on Linux and Mac, the stdlib will usually be somewhere that's not user-writable

On Windows, this depends on the install location. Perhaps there should be an option for edit-save or view only to avoid accidental changes.

The problem is that, if the standard way for users to see stdlib sources is to copy them from somewhere else (like $install/src/Lib) into a stdlib directory (like $install/Lib), then that stdlib directory has to be writable--and on Mac and Linux, it's not.

On 04.02.2016 14:09, Nick Coghlan wrote:

On 2 February 2016 at 06:39, Andrew Barnert via Python-Dev wrote:

...

On Feb 1, 2016, at 09:59, mike.romberg@comcast.net wrote:

...

If the stdlib were to use implicit namespace packages ( https://www.python.org/dev/peps/pep-0420/ ) and the various loaders/importers as well, then python could do what I've done with an embedded python application for years. Freeze the stdlib (or put it in a zipfile or whatever is fast). Then arrange PYTHONPATH to first look on the filesystem and then look in the frozen/ziped storage. This is a great solution for experienced developers, but I think it would be pretty bad for novices or transplants from other languages (maybe even including Python 2).

There are already multiple duplicate questions every month on StackOverflow from people asking "how do I find the source to stdlib module X". The canonical answer starts off by explaining how to import the module and use its __file__, which everyone is able to handle. If we have to instead explain how to work out the .py name from the qualified module name, how to work out the stdlib path from sys.path, and then how to find the source from those two things, with the caveat that it may not be installed at all on some platforms, and how to make sure what they're asking about really is a stdlib module, and how to make sure they aren't shadowing it with a module elsewhere on sys.path, that's a lot more complicated. Especially when you consider that some people on Windows and Mac are writing Python scripts without ever learning how to use the terminal or find their Python packages via Explorer/Finder. For folks that *do* know how to use the terminal:

$ python3 -m inspect --details inspect Target: inspect Origin: /usr/lib64/python3.4/inspect.py Cached: /usr/lib64/python3.4/__pycache__/inspect.cpython-34.pyc Loader: <_frozen_importlib.SourceFileLoader object at 0x7f0d8d23d9b0>

(And if they just want to *read* the source code, then leaving out "--details" prints the full module source, and would work even if the standard library were in a zip archive)

I want to see and debug also core Python in PyCharm and this is not acceptable. If you want to make it opt-in, fine. But opt-out is a no-go. I have a side-by-side comparison as we use Java and Python in production. It's the *ease of access* that makes Python great compared to Java. @Andrew Even for experienced developers it just sucks and there are more important things to do. Best, Sven

On Thu, Feb 04, 2016 at 07:58:30PM -0500, Terry Reedy wrote:

...

...

For folks that *do* know how to use the terminal:

$ python3 -m inspect --details inspect Target: inspect Origin: /usr/lib64/python3.4/inspect.py Cached: /usr/lib64/python3.4/__pycache__/inspect.cpython-34.pyc Loader: <_frozen_importlib.SourceFileLoader object at 0x7f0d8d23d9b0>

(And if they just want to *read* the source code, then leaving out "--details" prints the full module source, and would work even if the standard library were in a zip archive)

This is completely inadequate as a replacement for loading source into an editor, even if just for reading. [...]

I agree with Terry. The inspect trick Nick describes above is a great feature to have, but it's not a substitute for opening the source in an editor, not even on OSes where the command line tools are more powerful than Windows' default tools. [...]

I agree that removing stdlib python source files by default is an poor idea. The disk space saved is trivial. So, for me, would be nearly all of the time saving.

I too would be very reluctant to remove the source files from Python by default, but I have an alternative. I don't know if this is a ridiculous idea or not, but now that the .pyc bytecode files are kept in a separate __pycache__ directory, could we freeze that directory and leave the source files available for reading? (I'm not even sure if this suggestion makes sense, since I'm not really sure what "freezing" the stdlib entails. Is it documented anywhere?) -- Steve

On 02/01/2016 08:40 AM, R. David Murray wrote:

On Mon, 01 Feb 2016 14:12:27 +1100, Steven D'Aprano wrote:

...

I find that being able to easily open stdlib .py files in a text editor to read the source is extremely valuable. I've learned much more from reading the source than from (e.g.) StackOverflow. Likewise, it's often handy to do a grep over the stdlib. When you talk about freezing the stdlib, what exactly does that mean?

- will the source files still be there?

Well, Brett said it would be optional, though perhaps the above paragraph is asking about doing it in our Windows build. But the linux distros might make also use the option if it exists, so the question is very meaningful. However, you'd have to ask the distro if the source would be shipped in the linux case, and I'd guess not in most cases.

I don't know about anyone else, but on my own development systems it is not that unusual for me to *edit* the stdlib files (to add debug prints) while debugging my own programs. Freeze would definitely interfere with that. I could, of course, install a separate source build on my dev system, but I thought it worth mentioning as a factor.

Yup, so do I.

On the other hand, if the distros go the way Nick has (I think) been advocating, and have a separate 'system python for system scripts' that is independent of the one installed for user use, having the system-only python be frozen and sourceless would actually make sense on a couple of levels.

Agreed. -- ~Ethan~

On 2/1/2016 9:20 AM, Ethan Furman wrote:

...

On 02/01/2016 08:40 AM, R. David Murray wrote: <snip>

...

On the other hand, if the distros go the way Nick has (I think) been advocating, and have a separate 'system python for system scripts' that is independent of the one installed for user use, having the system-only python be frozen and sourceless would actually make sense on a couple of levels.

Agreed.

Except for that nasty licensing issue requiring source code.

Emile Licensing requires, in the GPL at least, that the *modified* sources be made *available*, not that they be shipped with the product. Looking at

On 2/5/2016 12:27, Emile van Sebille wrote: the Python license, and what tools already do, there is zero need to ship the source to stay compliant.

On Fri, 5 Feb 2016 at 10:34 Emile van Sebille wrote:

On 2/5/2016 9:37 AM, Alexander Walters wrote:

...

On 2/5/2016 12:27, Emile van Sebille wrote:

...

On 2/1/2016 9:20 AM, Ethan Furman wrote:

...

On 02/01/2016 08:40 AM, R. David Murray wrote: <snip>

...

On the other hand, if the distros go the way Nick has (I think) been advocating, and have a separate 'system python for system scripts'

that

...

...

...

...

is independent of the one installed for user use, having the system-only python be frozen and sourceless would actually make sense on a couple of levels.

Agreed.

Except for that nasty licensing issue requiring source code.

Emile Licensing requires, in the GPL at least, that the *modified* sources be made *available*, not that they be shipped with the product. Looking at the Python license, and what tools already do, there is zero need to ship the source to stay compliant.

Hmm, the annotated Open Source Definition explicitly states "The program must include source code" -- how did I misinterpret that?

Because you left off the part following: "... and must allow distribution in source code as well as compiled form". This is entirely a discussion of distribution in a compiled form.

On Friday, February 5, 2016 11:57 AM, Emile van Sebille wrote:

Aah, 'must' is less restrictive in this context than I expected. When you combine the two halves the first part might be more accurately phrased as 'The program must make source code available' rather than 'must include' which I understood to mean 'ship with'.

First, step back and think of this in common sense terms: If being open source required any Python installation to have the .py source to the .pyc or .zip files in the stdlib, surely it would also require any Python installation to have the .c source to the interpreter too. But lots of people have Python without having the .c source. Also, the GPL isn't typical of all open source licenses, it's only typical of _copyleft_ licenses. Permissive licenses, like Python's, are very different. Copyleft licenses are designed to make sure that all derived works are also copylefted; permissive licenses are designed to permit derived works as widely as possible. As the Python license specifically says, "All Python licenses, unlike the GPL, let you distribute a modified version without making your changes open source." Meanwhile, the fact that someone has decided that the Python license qualifies under the Open Source Definition doesn't mean the OSD is the right way to understand it. Read the license itself, or one of the summaries at opensource.org or fsf.org. (And if you still can't figure something out, and it's important to your work, you almost certainly need to ask a lawyer.) So, if you think the first sentence of section 2 of the OSD contradicts the explanation in the rest of the paragraph--well, even if you're right, that doesn't affect Python's license at all. Finally, if you want to see what it takes to actually make all the terms unambiguous both to ordinary human beings and to legal codes, see the GPL FAQ sections on their definitions of "propagate" and "convey". It may take you lots of careful reading to understand it, but when you finally do, it's definitely unambiguous.

Executive summary: There is no licensing issue because Python isn't copyleft. Stick to the pragmatic *technical* issue of how to reliably provide corresponding source to those who want to look at that source (just because that's how we do things in Python). Emile van Sebille writes:

Except for that nasty licensing issue requiring source code.

CPython is not now and never has been copyleft. CPython is distributed by the PSF *as* open source with a license that *permits* redistribution of original source and derivatives (including executables), but legally need not *remain* open source downstream. The remaining issue is the PSF's CLA which permits the PSF to relicense/sublicense under any open source license. However it's not clear to me that the PSF is required by the CLA to distribute source! It receives the code under very permissive licenses, and the CLA merely names the contributor's chosen license. I imagine those licenses determine whether the PSF must distribute source. If so, no, not even the PSF is bound (legally) to distribute Python source. Of course if *you* want to you can GPL Python (I think that's now possible, at one time there was a issue with the CNRI license IIRC), and then licensees of *your* distribution (but not you!) are required to distribute source. Of course our trust in the PSF is based on the moral principle of reciprocity: we contribute to the PSF's distribution as open source (according to the CLA) in large part because we expect to receive open source back. But if the PSF ever goes so wrong as to even think of taking advantage of that loophole, we are well and truly hosed anyway. (Among other things, that means a voting majority of the current PSF Board -- many of them core developers -- fell under a bus.) So don't worry about it.

Chris Angelico writes:

And even the GPL doesn't require you to distribute the source along with every copy of the binary. As long as the source is *available*, it's acceptable to distribute just the binary for convenience.

True (and it would apply to frozen Python as long as the source includes the build scripts such as setup.py used to "freeze" Python), but it can be complex (especially for commercial distribution). However, the technical problem remains. For example, you mention Debian. While Debian keeps its source and binary packages very close to "in sync" on the server, there are several gotchas. For example, Debian does not restrict itself to packaging patches, it sometimes breaks your security when it thinks it's smarter than Bruce. So ... is the corresponding source you're interested in the patched or unpatched source? Do you know which you get when you install the source package? Do you know how to get the other? Suppose for reasons of stability you've "pinned" the binary. Is the corresponding Debian source package still easily available? Did you think of that gotcha when you installed the source package, or did you just assume they were still in sync? I'm sure somebody with the "security mindset" (eg, Bruce) can think of many more.... It's not Python's responsibility to solve these gotchas, of course. Many (eg, do you want patched vs. unpatched) are use-case-dependent anyway. However, many of them do go away (and Python has fulfilled any imaginable responsibility) if we distribute source with the binaries, or arrange that binaries are built from source at installation.

On Feb 06, 2016, at 04:38 PM, Chris Angelico wrote:

Right, sure. The technical problems are still there. Although I'm fairly confident that Debian's binaries would correspond to Debian's source - but honestly, if I'm looking for sources for anything other than the kernel, I probably want to get the latest from source control, rather than using the somewhat older version shipped in the repos.

As to availability, though, most of the big distros (including Debian) keep their sources around for a long time.

Not to get too deep into what other projects do, but yes in Debian, you can always get the patched source that corresponds to the binary you've installed, usually in both version controlled form and otherwise. I'd expect this to be true of most if not all of the Linux distros. A more interesting question is how you can actually verify this equivalence, and there are folks across the ecosystem working on reproducible builds. The idea is that you should be able to take the source that *claims* to correspond to that binary, and using the established build tools, locally reproduce a bit-wise exact duplicate of the binary. I've applied and submitted several patches to various upstreams that help with this effort, such as being able to pass in "locked" datetimes instead of the package always using e.g. datetime.now(). Let's not dive down the rabbit hole too far into how you can trust your build tool chain, and every other layer down to the quantum. Cheers, -Barry

On 2 February 2016 at 02:40, R. David Murray wrote:

On the other hand, if the distros go the way Nick has (I think) been advocating, and have a separate 'system python for system scripts' that is independent of the one installed for user use, having the system-only python be frozen and sourceless would actually make sense on a couple of levels.

While omitting Python source files does let us reduce base image sizes (quite significantly), the current perspective in Fedora and Project Atomic is that going bytecode-only (whether frozen or not) breaks too many things to be worthwhile. As one simple example, it means tracebacks no longer include source code lines, dramatically increasing the difficulty of debugging failures. As such, we're more likely to pursue minimisation efforts by splitting the standard library up into "stuff essential distro components use" and "the rest of the standard library that upstream defines" than by figuring out how to avoid shipping source files (I believe Debian already makes this distinction with the python-minimal vs python split). Zipping up the standard library doesn't break tracebacks though, so it's potentially worth exploring that option further. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Sat, Jan 30, 2016, 12:30 Sven R. Kunze wrote:

On 30.01.2016 19:20, Serhiy Storchaka wrote:

...

AFAIK the most time is spent in system calls like stat or open. Archiving the stdlib into the ZIP file and using zipimport can decrease Python startup time (perhaps there is an open issue about this).

Oh, please don't. One thing I love about Python is the ease of access.

It wouldn't be a requirement, just a nootion

I personally think that startup time is not really a big issue; even when it comes to microbenchmarks.

You might not, but just about every command-line app does. -brett

Best, Sven _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org

Brett Cannon writes:

A lazy importer was added in Python 3.5 and it was not possible without the module spec refactoring.

Wow... Thank you, I didn't know about that. Now for the next question: how am I supposed to use it? The following documentation leaves me absolutely clueless: """This class only works with loaders that define exec_module() as control over what module type is used for the module is required. For those same reasons, the loader’s create_module() method will be ignored (i.e., the loader’s method should only return None). Finally, modules which substitute the object placed into sys.modules will not work as there is no way to properly replace the module references throughout the interpreter safely; ValueError is raised if such a substitution is detected.""" (reference: https://docs.python.org/3/library/importlib.html#importlib.util.LazyLoader) I want to import lazily the modules from package "foobar.*", but not other modules as other libraries may depend on import side effects. How do I do that? The quoted snippet doesn't really help. Regards Antoine.

On Sun, 31 Jan 2016, 15:36 Terry Reedy wrote:

On 1/31/2016 12:09 PM, Antoine Pitrou wrote:

...

The following documentation leaves me absolutely clueless:

"""This class only works with loaders that define exec_module() as control over what module type is used for the module is required.

No wonder. I cannot parse it as an English sentence. It needs rewriting.

Feel free to open an issue to clarify the wording. -Brett

...

For those same reasons, the loader’s create_module() method will be ignored (i.e., the loader’s method should only return None). Finally, modules which substitute the object placed into sys.modules will not work as there is no way to properly replace the module references throughout the interpreter safely; ValueError is raised if such a substitution is detected."""

(reference:

https://docs.python.org/3/library/importlib.html#importlib.util.LazyLoader )

-- Terry Jan Reedy

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org

There are no example docs for it yet, but enough people have asked this week about how to set up a custom importer that I will write up a generic example case which will make sense for a lazy loader (need to file the issue before I forget). On Sun, 31 Jan 2016, 09:11 Donald Stufft wrote:

On Jan 31, 2016, at 12:02 PM, Brett Cannon wrote:

A lazy importer was added in Python 3.5

Is there any docs on how to actually use the LazyLoader in 3.5? I can’t seem to find any but I don’t really know the import system that well.

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Thanks, Brett. Wasn't aware of lazy imports as well. I think that one is even better reducing startup time as freezing stdlib. On 31.01.2016 18:57, Brett Cannon wrote:

I have opened http://bugs.python.org/issue26252 to track writing the example (and before ppl go playing with the lazy loader, be aware of http://bugs.python.org/issue26186).

On Sun, 31 Jan 2016 at 09:26 Brett Cannon mailto:brett@python.org> wrote:

There are no example docs for it yet, but enough people have asked this week about how to set up a custom importer that I will write up a generic example case which will make sense for a lazy loader (need to file the issue before I forget).

On Sun, 31 Jan 2016, 09:11 Donald Stufft mailto:donald@stufft.io> wrote:

...

On Jan 31, 2016, at 12:02 PM, Brett Cannon mailto:brett@python.org> wrote:

A lazy importer was added in Python 3.5

Is there any docs on how to actually use the LazyLoader in 3.5? I can’t seem to find any but I don’t really know the import system that well.

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/srkunze%40mail.de