Mailman 3 December 2021 - Python-checkins

bpo-37295: More direct computation of power-of-two factor in math.comb (GH-30313)
by mdickinson 31 Dec '21

31 Dec '21

https://github.com/python/cpython/commit/0b58bac3e7877d722bdbd3c38913dba2cb… commit: 0b58bac3e7877d722bdbd3c38913dba2cb212f13 branch: main author: Mark Dickinson <mdickinson(a)enthought.com> committer: mdickinson <dickinsm(a)gmail.com> date: 2021-12-31T19:52:27Z summary: bpo-37295: More direct computation of power-of-two factor in math.comb (GH-30313) Co-authored-by: Serhiy Storchaka <storchaka(a)gmail.com> files: M Modules/mathmodule.c diff --git a/Modules/mathmodule.c b/Modules/mathmodule.c index c4a23b6514f6b..952c51304a9fc 100644 --- a/Modules/mathmodule.c +++ b/Modules/mathmodule.c @@ -3462,7 +3462,7 @@ Python code to generate the values: reduced_fac_odd_part = fac_odd_part % (2**64) print(f"{reduced_fac_odd_part:#018x}u") */ -static uint64_t reduced_factorial_odd_part[] = { +static const uint64_t reduced_factorial_odd_part[] = { 0x0000000000000001u, 0x0000000000000001u, 0x0000000000000001u, 0x0000000000000003u, 0x0000000000000003u, 0x000000000000000fu, 0x000000000000002du, 0x000000000000013bu, 0x000000000000013bu, 0x0000000000000b13u, 0x000000000000375fu, 0x0000000000026115u, @@ -3494,7 +3494,7 @@ Python code to generate the values: inverted_fac_odd_part = pow(fac_odd_part, -1, 2**64) print(f"{inverted_fac_odd_part:#018x}u") */ -static uint64_t inverted_factorial_odd_part[] = { +static const uint64_t inverted_factorial_odd_part[] = { 0x0000000000000001u, 0x0000000000000001u, 0x0000000000000001u, 0xaaaaaaaaaaaaaaabu, 0xaaaaaaaaaaaaaaabu, 0xeeeeeeeeeeeeeeefu, 0x4fa4fa4fa4fa4fa5u, 0x2ff2ff2ff2ff2ff3u, 0x2ff2ff2ff2ff2ff3u, 0x938cc70553e3771bu, 0xb71c27cddd93e49fu, 0xb38e3229fcdee63du, @@ -3514,6 +3514,25 @@ static uint64_t inverted_factorial_odd_part[] = { 0x547fb1b8ab9d0ba3u, 0x8f15a826498852e3u, 0x32e1a03f38880283u, 0x3de4cce63283f0c1u, }; +/* exponent of the largest power of 2 dividing factorial(n), for n in range(68) + +Python code to generate the values: + +import math + +for n in range(68): + fac = math.factorial(n) + fac_trailing_zeros = (fac & -fac).bit_length() - 1 + print(fac_trailing_zeros) +*/ + +static const uint8_t factorial_trailing_zeros[] = { + 0, 0, 1, 1, 3, 3, 4, 4, 7, 7, 8, 8, 10, 10, 11, 11, // 0-15 + 15, 15, 16, 16, 18, 18, 19, 19, 22, 22, 23, 23, 25, 25, 26, 26, // 16-31 + 31, 31, 32, 32, 34, 34, 35, 35, 38, 38, 39, 39, 41, 41, 42, 42, // 32-47 + 46, 46, 47, 47, 49, 49, 50, 50, 53, 53, 54, 54, 56, 56, 57, 57, // 48-63 + 63, 63, 64, 64, // 64-67 +}; /*[clinic input] math.comb @@ -3588,15 +3607,14 @@ math_comb_impl(PyObject *module, PyObject *n, PyObject *k) where 2**shift is the largest power of two dividing comb(n, k) and comb_odd_part is comb(n, k) >> shift. comb_odd_part can be calculated efficiently via arithmetic modulo 2**64, using three - lookups and two uint64_t multiplications, while the necessary - shift can be computed via Kummer's theorem: it's the number of - carries when adding k to n - k in binary, which in turn is the - number of set bits of n ^ k ^ (n - k). + lookups and two uint64_t multiplications. */ uint64_t comb_odd_part = reduced_factorial_odd_part[ni] * inverted_factorial_odd_part[ki] * inverted_factorial_odd_part[ni - ki]; - int shift = _Py_popcount32((uint32_t)(ni ^ ki ^ (ni - ki))); + int shift = factorial_trailing_zeros[ni] + - factorial_trailing_zeros[ki] + - factorial_trailing_zeros[ni - ki]; result = PyLong_FromUnsignedLongLong(comb_odd_part << shift); goto done; }

1 0

bpo-45321: Add missing error codes to module `xml.parsers.expat.errors` (GH-30188)
by scoder 31 Dec '21

31 Dec '21

https://github.com/python/cpython/commit/e18d81569fa0564f3bc7bcfd2fce26ec91… commit: e18d81569fa0564f3bc7bcfd2fce26ec91ba0a6e branch: main author: Sebastian Pipping <sebastian(a)pipping.org> committer: scoder <stefan_ml(a)behnel.de> date: 2021-12-31T10:57:00+01:00 summary: bpo-45321: Add missing error codes to module `xml.parsers.expat.errors` (GH-30188) The idea is to ensure that module `xml.parsers.expat.errors` contains all known error codes and messages, even when CPython is compiled or run with an outdated version of libexpat. https://bugs.python.org/issue45321 files: A Misc/NEWS.d/next/Library/2021-12-19-00-00-48.bpo-45321.OyuhaY.rst M Doc/library/pyexpat.rst M Modules/pyexpat.c diff --git a/Doc/library/pyexpat.rst b/Doc/library/pyexpat.rst index 034e579315de0..d6581e21b01c0 100644 --- a/Doc/library/pyexpat.rst +++ b/Doc/library/pyexpat.rst @@ -867,6 +867,40 @@ The ``errors`` module has the following attributes: .. data:: XML_ERROR_SUSPEND_PE +.. data:: XML_ERROR_RESERVED_PREFIX_XML + + An attempt was made to + undeclare reserved namespace prefix ``xml`` + or to bind it to another namespace URI. + + +.. data:: XML_ERROR_RESERVED_PREFIX_XMLNS + + An attempt was made to declare or undeclare reserved namespace prefix ``xmlns``. + + +.. data:: XML_ERROR_RESERVED_NAMESPACE_URI + + An attempt was made to bind the URI of one the reserved namespace + prefixes ``xml`` and ``xmlns`` to another namespace prefix. + + +.. data:: XML_ERROR_INVALID_ARGUMENT + + This should not be reported to Python applications. + + +.. data:: XML_ERROR_NO_BUFFER + + This should not be reported to Python applications. + + +.. data:: XML_ERROR_AMPLIFICATION_LIMIT_BREACH + + The limit on input amplification factor (from DTD and entities) + has been breached. + + .. rubric:: Footnotes .. [1] The encoding string included in XML output should conform to the diff --git a/Misc/NEWS.d/next/Library/2021-12-19-00-00-48.bpo-45321.OyuhaY.rst b/Misc/NEWS.d/next/Library/2021-12-19-00-00-48.bpo-45321.OyuhaY.rst new file mode 100644 index 0000000000000..171bf8a43e645 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-12-19-00-00-48.bpo-45321.OyuhaY.rst @@ -0,0 +1 @@ +Added missing error codes to module ``xml.parsers.expat.errors``. diff --git a/Modules/pyexpat.c b/Modules/pyexpat.c index b3d9bdda7e7ac..f2baab757f90c 100644 --- a/Modules/pyexpat.c +++ b/Modules/pyexpat.c @@ -1650,16 +1650,95 @@ add_submodule(PyObject *mod, const char *fullname) return submodule; } +struct ErrorInfo { + const char * name; /* Error constant name, e.g. "XML_ERROR_NO_MEMORY" */ + const char * description; /* Error description as returned by XML_ErrorString(<int>) */ +}; + +static +struct ErrorInfo error_info_of[] = { + {NULL, NULL}, /* XML_ERROR_NONE (value 0) is not exposed */ + + {"XML_ERROR_NO_MEMORY", "out of memory"}, + {"XML_ERROR_SYNTAX", "syntax error"}, + {"XML_ERROR_NO_ELEMENTS", "no element found"}, + {"XML_ERROR_INVALID_TOKEN", "not well-formed (invalid token)"}, + {"XML_ERROR_UNCLOSED_TOKEN", "unclosed token"}, + {"XML_ERROR_PARTIAL_CHAR", "partial character"}, + {"XML_ERROR_TAG_MISMATCH", "mismatched tag"}, + {"XML_ERROR_DUPLICATE_ATTRIBUTE", "duplicate attribute"}, + {"XML_ERROR_JUNK_AFTER_DOC_ELEMENT", "junk after document element"}, + {"XML_ERROR_PARAM_ENTITY_REF", "illegal parameter entity reference"}, + {"XML_ERROR_UNDEFINED_ENTITY", "undefined entity"}, + {"XML_ERROR_RECURSIVE_ENTITY_REF", "recursive entity reference"}, + {"XML_ERROR_ASYNC_ENTITY", "asynchronous entity"}, + {"XML_ERROR_BAD_CHAR_REF", "reference to invalid character number"}, + {"XML_ERROR_BINARY_ENTITY_REF", "reference to binary entity"}, + {"XML_ERROR_ATTRIBUTE_EXTERNAL_ENTITY_REF", "reference to external entity in attribute"}, + {"XML_ERROR_MISPLACED_XML_PI", "XML or text declaration not at start of entity"}, + {"XML_ERROR_UNKNOWN_ENCODING", "unknown encoding"}, + {"XML_ERROR_INCORRECT_ENCODING", "encoding specified in XML declaration is incorrect"}, + {"XML_ERROR_UNCLOSED_CDATA_SECTION", "unclosed CDATA section"}, + {"XML_ERROR_EXTERNAL_ENTITY_HANDLING", "error in processing external entity reference"}, + {"XML_ERROR_NOT_STANDALONE", "document is not standalone"}, + {"XML_ERROR_UNEXPECTED_STATE", "unexpected parser state - please send a bug report"}, + {"XML_ERROR_ENTITY_DECLARED_IN_PE", "entity declared in parameter entity"}, + {"XML_ERROR_FEATURE_REQUIRES_XML_DTD", "requested feature requires XML_DTD support in Expat"}, + {"XML_ERROR_CANT_CHANGE_FEATURE_ONCE_PARSING", "cannot change setting once parsing has begun"}, + + /* Added in Expat 1.95.7. */ + {"XML_ERROR_UNBOUND_PREFIX", "unbound prefix"}, + + /* Added in Expat 1.95.8. */ + {"XML_ERROR_UNDECLARING_PREFIX", "must not undeclare prefix"}, + {"XML_ERROR_INCOMPLETE_PE", "incomplete markup in parameter entity"}, + {"XML_ERROR_XML_DECL", "XML declaration not well-formed"}, + {"XML_ERROR_TEXT_DECL", "text declaration not well-formed"}, + {"XML_ERROR_PUBLICID", "illegal character(s) in public id"}, + {"XML_ERROR_SUSPENDED", "parser suspended"}, + {"XML_ERROR_NOT_SUSPENDED", "parser not suspended"}, + {"XML_ERROR_ABORTED", "parsing aborted"}, + {"XML_ERROR_FINISHED", "parsing finished"}, + {"XML_ERROR_SUSPEND_PE", "cannot suspend in external parameter entity"}, + + /* Added in 2.0.0. */ + {"XML_ERROR_RESERVED_PREFIX_XML", "reserved prefix (xml) must not be undeclared or bound to another namespace name"}, + {"XML_ERROR_RESERVED_PREFIX_XMLNS", "reserved prefix (xmlns) must not be declared or undeclared"}, + {"XML_ERROR_RESERVED_NAMESPACE_URI", "prefix must not be bound to one of the reserved namespace names"}, + + /* Added in 2.2.1. */ + {"XML_ERROR_INVALID_ARGUMENT", "invalid argument"}, + + /* Added in 2.3.0. */ + {"XML_ERROR_NO_BUFFER", "a successful prior call to function XML_GetBuffer is required"}, + + /* Added in 2.4.0. */ + {"XML_ERROR_AMPLIFICATION_LIMIT_BREACH", "limit on input amplification factor (from DTD and entities) breached"} +}; + static int add_error(PyObject *errors_module, PyObject *codes_dict, - PyObject *rev_codes_dict, const char *name, int value) + PyObject *rev_codes_dict, size_t error_index) { - const char *error_string = XML_ErrorString(value); + const char * const name = error_info_of[error_index].name; + const int error_code = (int)error_index; + + /* NOTE: This keeps the source of truth regarding error + * messages with libexpat and (by definiton) in bulletproof sync + * with the other uses of the XML_ErrorString function + * elsewhere within this file. pyexpat's copy of the messages + * only acts as a fallback in case of outdated runtime libexpat, + * where it returns NULL. */ + const char *error_string = XML_ErrorString(error_code); + if (error_string == NULL) { + error_string = error_info_of[error_index].description; + } + if (PyModule_AddStringConstant(errors_module, name, error_string) < 0) { return -1; } - PyObject *num = PyLong_FromLong(value); + PyObject *num = PyLong_FromLong(error_code); if (num == NULL) { return -1; } @@ -1699,53 +1778,16 @@ add_errors_module(PyObject *mod) goto error; } -#define ADD_CONST(name) do { \ - if (add_error(errors_module, codes_dict, rev_codes_dict, \ - #name, name) < 0) { \ - goto error; \ - } \ - } while(0) + size_t error_index = 0; + for (; error_index < sizeof(error_info_of) / sizeof(struct ErrorInfo); error_index++) { + if (error_info_of[error_index].name == NULL) { + continue; + } - ADD_CONST(XML_ERROR_NO_MEMORY); - ADD_CONST(XML_ERROR_SYNTAX); - ADD_CONST(XML_ERROR_NO_ELEMENTS); - ADD_CONST(XML_ERROR_INVALID_TOKEN); - ADD_CONST(XML_ERROR_UNCLOSED_TOKEN); - ADD_CONST(XML_ERROR_PARTIAL_CHAR); - ADD_CONST(XML_ERROR_TAG_MISMATCH); - ADD_CONST(XML_ERROR_DUPLICATE_ATTRIBUTE); - ADD_CONST(XML_ERROR_JUNK_AFTER_DOC_ELEMENT); - ADD_CONST(XML_ERROR_PARAM_ENTITY_REF); - ADD_CONST(XML_ERROR_UNDEFINED_ENTITY); - ADD_CONST(XML_ERROR_RECURSIVE_ENTITY_REF); - ADD_CONST(XML_ERROR_ASYNC_ENTITY); - ADD_CONST(XML_ERROR_BAD_CHAR_REF); - ADD_CONST(XML_ERROR_BINARY_ENTITY_REF); - ADD_CONST(XML_ERROR_ATTRIBUTE_EXTERNAL_ENTITY_REF); - ADD_CONST(XML_ERROR_MISPLACED_XML_PI); - ADD_CONST(XML_ERROR_UNKNOWN_ENCODING); - ADD_CONST(XML_ERROR_INCORRECT_ENCODING); - ADD_CONST(XML_ERROR_UNCLOSED_CDATA_SECTION); - ADD_CONST(XML_ERROR_EXTERNAL_ENTITY_HANDLING); - ADD_CONST(XML_ERROR_NOT_STANDALONE); - ADD_CONST(XML_ERROR_UNEXPECTED_STATE); - ADD_CONST(XML_ERROR_ENTITY_DECLARED_IN_PE); - ADD_CONST(XML_ERROR_FEATURE_REQUIRES_XML_DTD); - ADD_CONST(XML_ERROR_CANT_CHANGE_FEATURE_ONCE_PARSING); - /* Added in Expat 1.95.7. */ - ADD_CONST(XML_ERROR_UNBOUND_PREFIX); - /* Added in Expat 1.95.8. */ - ADD_CONST(XML_ERROR_UNDECLARING_PREFIX); - ADD_CONST(XML_ERROR_INCOMPLETE_PE); - ADD_CONST(XML_ERROR_XML_DECL); - ADD_CONST(XML_ERROR_TEXT_DECL); - ADD_CONST(XML_ERROR_PUBLICID); - ADD_CONST(XML_ERROR_SUSPENDED); - ADD_CONST(XML_ERROR_NOT_SUSPENDED); - ADD_CONST(XML_ERROR_ABORTED); - ADD_CONST(XML_ERROR_FINISHED); - ADD_CONST(XML_ERROR_SUSPEND_PE); -#undef ADD_CONST + if (add_error(errors_module, codes_dict, rev_codes_dict, error_index) < 0) { + goto error; + } + } if (PyModule_AddStringConstant(errors_module, "__doc__", "Constants used to describe "

1 0

bpo-46118: Make sure importlib.resources is included. (GH-30311)
by jaraco 31 Dec '21

31 Dec '21

https://github.com/python/cpython/commit/35628e4cde71e54afe12aea50c74069afe… commit: 35628e4cde71e54afe12aea50c74069afe2c3389 branch: main author: Jason R. Coombs <jaraco(a)jaraco.com> committer: jaraco <jaraco(a)jaraco.com> date: 2021-12-30T22:39:24-05:00 summary: bpo-46118: Make sure importlib.resources is included. (GH-30311) files: M Makefile.pre.in diff --git a/Makefile.pre.in b/Makefile.pre.in index ed77bebfab901..fbd4c3a23fd81 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1911,7 +1911,7 @@ LIBSUBDIRS= asyncio \ html \ http \ idlelib idlelib/Icons \ - importlib importlib/metadata \ + importlib importlib/resources importlib/metadata \ json \ lib2to3 lib2to3/fixes lib2to3/pgen2 \ logging \

1 0

bpo-46109: Separate out files relating to importlib.resources (GH-30160)
by jaraco 31 Dec '21

31 Dec '21

https://github.com/python/cpython/commit/99945c6b5cf280bd90075cffae942af449… commit: 99945c6b5cf280bd90075cffae942af44941abcc branch: main author: Jason R. Coombs <jaraco(a)jaraco.com> committer: jaraco <jaraco(a)jaraco.com> date: 2021-12-30T21:17:05-05:00 summary: bpo-46109: Separate out files relating to importlib.resources (GH-30160) * Separate out files relating to importlib.resources * Update Introduction to direct readers to the submodule documentation. * Create separate file for abcs relating to resources. * Move abc docs back to where they were. files: A Doc/library/importlib.resources.abc.rst A Doc/library/importlib.resources.rst A Misc/NEWS.d/next/Documentation/2021-12-16-21-13-55.bpo-46109.0-RNzu.rst M Doc/library/importlib.rst M Doc/library/modules.rst diff --git a/Doc/library/importlib.resources.abc.rst b/Doc/library/importlib.resources.abc.rst new file mode 100644 index 0000000000000..6b0e1e04c8852 --- /dev/null +++ b/Doc/library/importlib.resources.abc.rst @@ -0,0 +1,385 @@ +.. class:: ResourceReader + + *Superseded by TraversableResources* + + An :term:`abstract base class` to provide the ability to read + *resources*. + + From the perspective of this ABC, a *resource* is a binary + artifact that is shipped within a package. Typically this is + something like a data file that lives next to the ``__init__.py`` + file of the package. The purpose of this class is to help abstract + out the accessing of such data files so that it does not matter if + the package and its data file(s) are stored in a e.g. zip file + versus on the file system. + + For any of methods of this class, a *resource* argument is + expected to be a :term:`path-like object` which represents + conceptually just a file name. This means that no subdirectory + paths should be included in the *resource* argument. This is + because the location of the package the reader is for, acts as the + "directory". Hence the metaphor for directories and file + names is packages and resources, respectively. This is also why + instances of this class are expected to directly correlate to + a specific package (instead of potentially representing multiple + packages or a module). + + Loaders that wish to support resource reading are expected to + provide a method called ``get_resource_reader(fullname)`` which + returns an object implementing this ABC's interface. If the module + specified by fullname is not a package, this method should return + :const:`None`. An object compatible with this ABC should only be + returned when the specified module is a package. + + .. versionadded:: 3.7 + + .. abstractmethod:: open_resource(resource) + + Returns an opened, :term:`file-like object` for binary reading + of the *resource*. + + If the resource cannot be found, :exc:`FileNotFoundError` is + raised. + + .. abstractmethod:: resource_path(resource) + + Returns the file system path to the *resource*. + + If the resource does not concretely exist on the file system, + raise :exc:`FileNotFoundError`. + + .. abstractmethod:: is_resource(name) + + Returns ``True`` if the named *name* is considered a resource. + :exc:`FileNotFoundError` is raised if *name* does not exist. + + .. abstractmethod:: contents() + + Returns an :term:`iterable` of strings over the contents of + the package. Do note that it is not required that all names + returned by the iterator be actual resources, e.g. it is + acceptable to return names for which :meth:`is_resource` would + be false. + + Allowing non-resource names to be returned is to allow for + situations where how a package and its resources are stored + are known a priori and the non-resource names would be useful. + For instance, returning subdirectory names is allowed so that + when it is known that the package and resources are stored on + the file system then those subdirectory names can be used + directly. + + The abstract method returns an iterable of no items. + + +.. class:: ResourceLoader + + An abstract base class for a :term:`loader` which implements the optional + :pep:`302` protocol for loading arbitrary resources from the storage + back-end. + + .. deprecated:: 3.7 + This ABC is deprecated in favour of supporting resource loading + through :class:`importlib.abc.ResourceReader`. + + .. abstractmethod:: get_data(path) + + An abstract method to return the bytes for the data located at *path*. + Loaders that have a file-like storage back-end + that allows storing arbitrary data + can implement this abstract method to give direct access + to the data stored. :exc:`OSError` is to be raised if the *path* cannot + be found. The *path* is expected to be constructed using a module's + :attr:`__file__` attribute or an item from a package's :attr:`__path__`. + + .. versionchanged:: 3.4 + Raises :exc:`OSError` instead of :exc:`NotImplementedError`. + + +.. class:: InspectLoader + + An abstract base class for a :term:`loader` which implements the optional + :pep:`302` protocol for loaders that inspect modules. + + .. method:: get_code(fullname) + + Return the code object for a module, or ``None`` if the module does not + have a code object (as would be the case, for example, for a built-in + module). Raise an :exc:`ImportError` if loader cannot find the + requested module. + + .. note:: + While the method has a default implementation, it is suggested that + it be overridden if possible for performance. + + .. index:: + single: universal newlines; importlib.abc.InspectLoader.get_source method + + .. versionchanged:: 3.4 + No longer abstract and a concrete implementation is provided. + + .. abstractmethod:: get_source(fullname) + + An abstract method to return the source of a module. It is returned as + a text string using :term:`universal newlines`, translating all + recognized line separators into ``'\n'`` characters. Returns ``None`` + if no source is available (e.g. a built-in module). Raises + :exc:`ImportError` if the loader cannot find the module specified. + + .. versionchanged:: 3.4 + Raises :exc:`ImportError` instead of :exc:`NotImplementedError`. + + .. method:: is_package(fullname) + + An optional method to return a true value if the module is a package, a + false value otherwise. :exc:`ImportError` is raised if the + :term:`loader` cannot find the module. + + .. versionchanged:: 3.4 + Raises :exc:`ImportError` instead of :exc:`NotImplementedError`. + + .. staticmethod:: source_to_code(data, path='<string>') + + Create a code object from Python source. + + The *data* argument can be whatever the :func:`compile` function + supports (i.e. string or bytes). The *path* argument should be + the "path" to where the source code originated from, which can be an + abstract concept (e.g. location in a zip file). + + With the subsequent code object one can execute it in a module by + running ``exec(code, module.__dict__)``. + + .. versionadded:: 3.4 + + .. versionchanged:: 3.5 + Made the method static. + + .. method:: exec_module(module) + + Implementation of :meth:`Loader.exec_module`. + + .. versionadded:: 3.4 + + .. method:: load_module(fullname) + + Implementation of :meth:`Loader.load_module`. + + .. deprecated:: 3.4 + use :meth:`exec_module` instead. + + +.. class:: ExecutionLoader + + An abstract base class which inherits from :class:`InspectLoader` that, + when implemented, helps a module to be executed as a script. The ABC + represents an optional :pep:`302` protocol. + + .. abstractmethod:: get_filename(fullname) + + An abstract method that is to return the value of :attr:`__file__` for + the specified module. If no path is available, :exc:`ImportError` is + raised. + + If source code is available, then the method should return the path to + the source file, regardless of whether a bytecode was used to load the + module. + + .. versionchanged:: 3.4 + Raises :exc:`ImportError` instead of :exc:`NotImplementedError`. + + +.. class:: FileLoader(fullname, path) + + An abstract base class which inherits from :class:`ResourceLoader` and + :class:`ExecutionLoader`, providing concrete implementations of + :meth:`ResourceLoader.get_data` and :meth:`ExecutionLoader.get_filename`. + + The *fullname* argument is a fully resolved name of the module the loader is + to handle. The *path* argument is the path to the file for the module. + + .. versionadded:: 3.3 + + .. attribute:: name + + The name of the module the loader can handle. + + .. attribute:: path + + Path to the file of the module. + + .. method:: load_module(fullname) + + Calls super's ``load_module()``. + + .. deprecated:: 3.4 + Use :meth:`Loader.exec_module` instead. + + .. abstractmethod:: get_filename(fullname) + + Returns :attr:`path`. + + .. abstractmethod:: get_data(path) + + Reads *path* as a binary file and returns the bytes from it. + + +.. class:: SourceLoader + + An abstract base class for implementing source (and optionally bytecode) + file loading. The class inherits from both :class:`ResourceLoader` and + :class:`ExecutionLoader`, requiring the implementation of: + + * :meth:`ResourceLoader.get_data` + * :meth:`ExecutionLoader.get_filename` + Should only return the path to the source file; sourceless + loading is not supported. + + The abstract methods defined by this class are to add optional bytecode + file support. Not implementing these optional methods (or causing them to + raise :exc:`NotImplementedError`) causes the loader to + only work with source code. Implementing the methods allows the loader to + work with source *and* bytecode files; it does not allow for *sourceless* + loading where only bytecode is provided. Bytecode files are an + optimization to speed up loading by removing the parsing step of Python's + compiler, and so no bytecode-specific API is exposed. + + .. method:: path_stats(path) + + Optional abstract method which returns a :class:`dict` containing + metadata about the specified path. Supported dictionary keys are: + + - ``'mtime'`` (mandatory): an integer or floating-point number + representing the modification time of the source code; + - ``'size'`` (optional): the size in bytes of the source code. + + Any other keys in the dictionary are ignored, to allow for future + extensions. If the path cannot be handled, :exc:`OSError` is raised. + + .. versionadded:: 3.3 + + .. versionchanged:: 3.4 + Raise :exc:`OSError` instead of :exc:`NotImplementedError`. + + .. method:: path_mtime(path) + + Optional abstract method which returns the modification time for the + specified path. + + .. deprecated:: 3.3 + This method is deprecated in favour of :meth:`path_stats`. You don't + have to implement it, but it is still available for compatibility + purposes. Raise :exc:`OSError` if the path cannot be handled. + + .. versionchanged:: 3.4 + Raise :exc:`OSError` instead of :exc:`NotImplementedError`. + + .. method:: set_data(path, data) + + Optional abstract method which writes the specified bytes to a file + path. Any intermediate directories which do not exist are to be created + automatically. + + When writing to the path fails because the path is read-only + (:attr:`errno.EACCES`/:exc:`PermissionError`), do not propagate the + exception. + + .. versionchanged:: 3.4 + No longer raises :exc:`NotImplementedError` when called. + + .. method:: get_code(fullname) + + Concrete implementation of :meth:`InspectLoader.get_code`. + + .. method:: exec_module(module) + + Concrete implementation of :meth:`Loader.exec_module`. + + .. versionadded:: 3.4 + + .. method:: load_module(fullname) + + Concrete implementation of :meth:`Loader.load_module`. + + .. deprecated:: 3.4 + Use :meth:`exec_module` instead. + + .. method:: get_source(fullname) + + Concrete implementation of :meth:`InspectLoader.get_source`. + + .. method:: is_package(fullname) + + Concrete implementation of :meth:`InspectLoader.is_package`. A module + is determined to be a package if its file path (as provided by + :meth:`ExecutionLoader.get_filename`) is a file named + ``__init__`` when the file extension is removed **and** the module name + itself does not end in ``__init__``. + + +.. class:: Traversable + + An object with a subset of pathlib.Path methods suitable for + traversing directories and opening files. + + .. versionadded:: 3.9 + + .. attribute:: name + + Abstract. The base name of this object without any parent references. + + .. abstractmethod:: iterdir() + + Yield Traversable objects in self. + + .. abstractmethod:: is_dir() + + Return True if self is a directory. + + .. abstractmethod:: is_file() + + Return True if self is a file. + + .. abstractmethod:: joinpath(child) + + Return Traversable child in self. + + .. abstractmethod:: __truediv__(child) + + Return Traversable child in self. + + .. abstractmethod:: open(mode='r', *args, **kwargs) + + *mode* may be 'r' or 'rb' to open as text or binary. Return a handle + suitable for reading (same as :attr:`pathlib.Path.open`). + + When opening as text, accepts encoding parameters such as those + accepted by :attr:`io.TextIOWrapper`. + + .. method:: read_bytes() + + Read contents of self as bytes. + + .. method:: read_text(encoding=None) + + Read contents of self as text. + + +.. class:: TraversableResources + + An abstract base class for resource readers capable of serving + the :meth:`importlib.resources.files` interface. Subclasses + :class:`importlib.abc.ResourceReader` and provides + concrete implementations of the :class:`importlib.abc.ResourceReader`'s + abstract methods. Therefore, any loader supplying + :class:`importlib.abc.TraversableReader` also supplies ResourceReader. + + Loaders that wish to support resource reading are expected to + implement this interface. + + .. versionadded:: 3.9 + + .. abstractmethod:: files() + + Returns a :class:`importlib.abc.Traversable` object for the loaded + package. diff --git a/Doc/library/importlib.resources.rst b/Doc/library/importlib.resources.rst new file mode 100644 index 0000000000000..f62d15dd6fdc9 --- /dev/null +++ b/Doc/library/importlib.resources.rst @@ -0,0 +1,185 @@ +:mod:`importlib.resources` -- Resources +--------------------------------------- + +.. module:: importlib.resources + :synopsis: Package resource reading, opening, and access + +**Source code:** :source:`Lib/importlib/resources.py` + +-------------- + +.. versionadded:: 3.7 + +This module leverages Python's import system to provide access to *resources* +within *packages*. If you can import a package, you can access resources +within that package. Resources can be opened or read, in either binary or +text mode. + +Resources are roughly akin to files inside directories, though it's important +to keep in mind that this is just a metaphor. Resources and packages **do +not** have to exist as physical files and directories on the file system. + +.. note:: + + This module provides functionality similar to `pkg_resources + <https://setuptools.readthedocs.io/en/latest/pkg_resources.html>`_ `Basic + Resource Access + <http://setuptools.readthedocs.io/en/latest/pkg_resources.html#basic-resourc…>`_ + without the performance overhead of that package. This makes reading + resources included in packages easier, with more stable and consistent + semantics. + + The standalone backport of this module provides more information + on `using importlib.resources + <http://importlib-resources.readthedocs.io/en/latest/using.html>`_ and + `migrating from pkg_resources to importlib.resources + <http://importlib-resources.readthedocs.io/en/latest/migration.html>`_ + and + `migrating legacy usage <https://importlib-resources.readthedocs.io/en/latest/using.html#migrating-f…>`_. + +Loaders that wish to support resource reading should implement a +``get_resource_reader(fullname)`` method as specified by +:class:`importlib.abc.ResourceReader`. + +The following types are defined. + +.. data:: Package + + The ``Package`` type is defined as ``Union[str, ModuleType]``. This means + that where the function describes accepting a ``Package``, you can pass in + either a string or a module. Module objects must have a resolvable + ``__spec__.submodule_search_locations`` that is not ``None``. + +.. data:: Resource + + This type describes the resource names passed into the various functions + in this package. This is defined as ``Union[str, os.PathLike]``. + + +The following functions are available. + + +.. function:: files(package) + + Returns an :class:`importlib.resources.abc.Traversable` object + representing the resource container for the package (think directory) + and its resources (think files). A Traversable may contain other + containers (think subdirectories). + + *package* is either a name or a module object which conforms to the + ``Package`` requirements. + + .. versionadded:: 3.9 + +.. function:: as_file(traversable) + + Given a :class:`importlib.resources.abc.Traversable` object representing + a file, typically from :func:`importlib.resources.files`, return + a context manager for use in a :keyword:`with` statement. + The context manager provides a :class:`pathlib.Path` object. + + Exiting the context manager cleans up any temporary file created when the + resource was extracted from e.g. a zip file. + + Use ``as_file`` when the Traversable methods + (``read_text``, etc) are insufficient and an actual file on + the file system is required. + + .. versionadded:: 3.9 + +.. function:: open_binary(package, resource) + + Open for binary reading the *resource* within *package*. + + *package* is either a name or a module object which conforms to the + ``Package`` requirements. *resource* is the name of the resource to open + within *package*; it may not contain path separators and it may not have + sub-resources (i.e. it cannot be a directory). This function returns a + ``typing.BinaryIO`` instance, a binary I/O stream open for reading. + + .. deprecated:: 3.11 + + +.. function:: open_text(package, resource, encoding='utf-8', errors='strict') + + Open for text reading the *resource* within *package*. By default, the + resource is opened for reading as UTF-8. + + *package* is either a name or a module object which conforms to the + ``Package`` requirements. *resource* is the name of the resource to open + within *package*; it may not contain path separators and it may not have + sub-resources (i.e. it cannot be a directory). *encoding* and *errors* + have the same meaning as with built-in :func:`open`. + + This function returns a ``typing.TextIO`` instance, a text I/O stream open + for reading. + + .. deprecated:: 3.11 + + +.. function:: read_binary(package, resource) + + Read and return the contents of the *resource* within *package* as + ``bytes``. + + *package* is either a name or a module object which conforms to the + ``Package`` requirements. *resource* is the name of the resource to open + within *package*; it may not contain path separators and it may not have + sub-resources (i.e. it cannot be a directory). This function returns the + contents of the resource as :class:`bytes`. + + .. deprecated:: 3.11 + + +.. function:: read_text(package, resource, encoding='utf-8', errors='strict') + + Read and return the contents of *resource* within *package* as a ``str``. + By default, the contents are read as strict UTF-8. + + *package* is either a name or a module object which conforms to the + ``Package`` requirements. *resource* is the name of the resource to open + within *package*; it may not contain path separators and it may not have + sub-resources (i.e. it cannot be a directory). *encoding* and *errors* + have the same meaning as with built-in :func:`open`. This function + returns the contents of the resource as :class:`str`. + + .. deprecated:: 3.11 + + +.. function:: path(package, resource) + + Return the path to the *resource* as an actual file system path. This + function returns a context manager for use in a :keyword:`with` statement. + The context manager provides a :class:`pathlib.Path` object. + + Exiting the context manager cleans up any temporary file created when the + resource needs to be extracted from e.g. a zip file. + + *package* is either a name or a module object which conforms to the + ``Package`` requirements. *resource* is the name of the resource to open + within *package*; it may not contain path separators and it may not have + sub-resources (i.e. it cannot be a directory). + + .. deprecated:: 3.11 + + +.. function:: is_resource(package, name) + + Return ``True`` if there is a resource named *name* in the package, + otherwise ``False``. Remember that directories are *not* resources! + *package* is either a name or a module object which conforms to the + ``Package`` requirements. + + .. deprecated:: 3.11 + + +.. function:: contents(package) + + Return an iterable over the named items within the package. The iterable + returns :class:`str` resources (e.g. files) and non-resources + (e.g. directories). The iterable does not recurse into subdirectories. + + *package* is either a name or a module object which conforms to the + ``Package`` requirements. + + .. deprecated:: 3.11 diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index 347e08e4a2846..23d908196669f 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -13,10 +13,13 @@ -------------- + Introduction ------------ -The purpose of the :mod:`importlib` package is two-fold. One is to provide the +The purpose of the :mod:`importlib` package is three-fold. + +One is to provide the implementation of the :keyword:`import` statement (and thus, by extension, the :func:`__import__` function) in Python source code. This provides an implementation of :keyword:`!import` which is portable to any Python @@ -27,6 +30,14 @@ Two, the components to implement :keyword:`import` are exposed in this package, making it easier for users to create their own custom objects (known generically as an :term:`importer`) to participate in the import process. +Three, the package contains modules exposing additional functionality for +managing aspects of Python packages: + +* :mod:`importlib.metadata` presents access to metadata from third-party + distributions. +* :mod:`importlib.resources` provides routines for accessing non-code + "resources" from Python packages. + .. seealso:: :ref:`import` @@ -495,578 +506,7 @@ ABC hierarchy:: The import machinery now takes care of this automatically. -.. class:: ResourceReader - - *Superseded by TraversableResources* - - An :term:`abstract base class` to provide the ability to read - *resources*. - - From the perspective of this ABC, a *resource* is a binary - artifact that is shipped within a package. Typically this is - something like a data file that lives next to the ``__init__.py`` - file of the package. The purpose of this class is to help abstract - out the accessing of such data files so that it does not matter if - the package and its data file(s) are stored in a e.g. zip file - versus on the file system. - - For any of methods of this class, a *resource* argument is - expected to be a :term:`path-like object` which represents - conceptually just a file name. This means that no subdirectory - paths should be included in the *resource* argument. This is - because the location of the package the reader is for, acts as the - "directory". Hence the metaphor for directories and file - names is packages and resources, respectively. This is also why - instances of this class are expected to directly correlate to - a specific package (instead of potentially representing multiple - packages or a module). - - Loaders that wish to support resource reading are expected to - provide a method called ``get_resource_reader(fullname)`` which - returns an object implementing this ABC's interface. If the module - specified by fullname is not a package, this method should return - :const:`None`. An object compatible with this ABC should only be - returned when the specified module is a package. - - .. versionadded:: 3.7 - - .. abstractmethod:: open_resource(resource) - - Returns an opened, :term:`file-like object` for binary reading - of the *resource*. - - If the resource cannot be found, :exc:`FileNotFoundError` is - raised. - - .. abstractmethod:: resource_path(resource) - - Returns the file system path to the *resource*. - - If the resource does not concretely exist on the file system, - raise :exc:`FileNotFoundError`. - - .. abstractmethod:: is_resource(name) - - Returns ``True`` if the named *name* is considered a resource. - :exc:`FileNotFoundError` is raised if *name* does not exist. - - .. abstractmethod:: contents() - - Returns an :term:`iterable` of strings over the contents of - the package. Do note that it is not required that all names - returned by the iterator be actual resources, e.g. it is - acceptable to return names for which :meth:`is_resource` would - be false. - - Allowing non-resource names to be returned is to allow for - situations where how a package and its resources are stored - are known a priori and the non-resource names would be useful. - For instance, returning subdirectory names is allowed so that - when it is known that the package and resources are stored on - the file system then those subdirectory names can be used - directly. - - The abstract method returns an iterable of no items. - - -.. class:: ResourceLoader - - An abstract base class for a :term:`loader` which implements the optional - :pep:`302` protocol for loading arbitrary resources from the storage - back-end. - - .. deprecated:: 3.7 - This ABC is deprecated in favour of supporting resource loading - through :class:`importlib.abc.ResourceReader`. - - .. abstractmethod:: get_data(path) - - An abstract method to return the bytes for the data located at *path*. - Loaders that have a file-like storage back-end - that allows storing arbitrary data - can implement this abstract method to give direct access - to the data stored. :exc:`OSError` is to be raised if the *path* cannot - be found. The *path* is expected to be constructed using a module's - :attr:`__file__` attribute or an item from a package's :attr:`__path__`. - - .. versionchanged:: 3.4 - Raises :exc:`OSError` instead of :exc:`NotImplementedError`. - - -.. class:: InspectLoader - - An abstract base class for a :term:`loader` which implements the optional - :pep:`302` protocol for loaders that inspect modules. - - .. method:: get_code(fullname) - - Return the code object for a module, or ``None`` if the module does not - have a code object (as would be the case, for example, for a built-in - module). Raise an :exc:`ImportError` if loader cannot find the - requested module. - - .. note:: - While the method has a default implementation, it is suggested that - it be overridden if possible for performance. - - .. index:: - single: universal newlines; importlib.abc.InspectLoader.get_source method - - .. versionchanged:: 3.4 - No longer abstract and a concrete implementation is provided. - - .. abstractmethod:: get_source(fullname) - - An abstract method to return the source of a module. It is returned as - a text string using :term:`universal newlines`, translating all - recognized line separators into ``'\n'`` characters. Returns ``None`` - if no source is available (e.g. a built-in module). Raises - :exc:`ImportError` if the loader cannot find the module specified. - - .. versionchanged:: 3.4 - Raises :exc:`ImportError` instead of :exc:`NotImplementedError`. - - .. method:: is_package(fullname) - - An optional method to return a true value if the module is a package, a - false value otherwise. :exc:`ImportError` is raised if the - :term:`loader` cannot find the module. - - .. versionchanged:: 3.4 - Raises :exc:`ImportError` instead of :exc:`NotImplementedError`. - - .. staticmethod:: source_to_code(data, path='<string>') - - Create a code object from Python source. - - The *data* argument can be whatever the :func:`compile` function - supports (i.e. string or bytes). The *path* argument should be - the "path" to where the source code originated from, which can be an - abstract concept (e.g. location in a zip file). - - With the subsequent code object one can execute it in a module by - running ``exec(code, module.__dict__)``. - - .. versionadded:: 3.4 - - .. versionchanged:: 3.5 - Made the method static. - - .. method:: exec_module(module) - - Implementation of :meth:`Loader.exec_module`. - - .. versionadded:: 3.4 - - .. method:: load_module(fullname) - - Implementation of :meth:`Loader.load_module`. - - .. deprecated:: 3.4 - use :meth:`exec_module` instead. - - -.. class:: ExecutionLoader - - An abstract base class which inherits from :class:`InspectLoader` that, - when implemented, helps a module to be executed as a script. The ABC - represents an optional :pep:`302` protocol. - - .. abstractmethod:: get_filename(fullname) - - An abstract method that is to return the value of :attr:`__file__` for - the specified module. If no path is available, :exc:`ImportError` is - raised. - - If source code is available, then the method should return the path to - the source file, regardless of whether a bytecode was used to load the - module. - - .. versionchanged:: 3.4 - Raises :exc:`ImportError` instead of :exc:`NotImplementedError`. - - -.. class:: FileLoader(fullname, path) - - An abstract base class which inherits from :class:`ResourceLoader` and - :class:`ExecutionLoader`, providing concrete implementations of - :meth:`ResourceLoader.get_data` and :meth:`ExecutionLoader.get_filename`. - - The *fullname* argument is a fully resolved name of the module the loader is - to handle. The *path* argument is the path to the file for the module. - - .. versionadded:: 3.3 - - .. attribute:: name - - The name of the module the loader can handle. - - .. attribute:: path - - Path to the file of the module. - - .. method:: load_module(fullname) - - Calls super's ``load_module()``. - - .. deprecated:: 3.4 - Use :meth:`Loader.exec_module` instead. - - .. abstractmethod:: get_filename(fullname) - - Returns :attr:`path`. - - .. abstractmethod:: get_data(path) - - Reads *path* as a binary file and returns the bytes from it. - - -.. class:: SourceLoader - - An abstract base class for implementing source (and optionally bytecode) - file loading. The class inherits from both :class:`ResourceLoader` and - :class:`ExecutionLoader`, requiring the implementation of: - - * :meth:`ResourceLoader.get_data` - * :meth:`ExecutionLoader.get_filename` - Should only return the path to the source file; sourceless - loading is not supported. - - The abstract methods defined by this class are to add optional bytecode - file support. Not implementing these optional methods (or causing them to - raise :exc:`NotImplementedError`) causes the loader to - only work with source code. Implementing the methods allows the loader to - work with source *and* bytecode files; it does not allow for *sourceless* - loading where only bytecode is provided. Bytecode files are an - optimization to speed up loading by removing the parsing step of Python's - compiler, and so no bytecode-specific API is exposed. - - .. method:: path_stats(path) - - Optional abstract method which returns a :class:`dict` containing - metadata about the specified path. Supported dictionary keys are: - - - ``'mtime'`` (mandatory): an integer or floating-point number - representing the modification time of the source code; - - ``'size'`` (optional): the size in bytes of the source code. - - Any other keys in the dictionary are ignored, to allow for future - extensions. If the path cannot be handled, :exc:`OSError` is raised. - - .. versionadded:: 3.3 - - .. versionchanged:: 3.4 - Raise :exc:`OSError` instead of :exc:`NotImplementedError`. - - .. method:: path_mtime(path) - - Optional abstract method which returns the modification time for the - specified path. - - .. deprecated:: 3.3 - This method is deprecated in favour of :meth:`path_stats`. You don't - have to implement it, but it is still available for compatibility - purposes. Raise :exc:`OSError` if the path cannot be handled. - - .. versionchanged:: 3.4 - Raise :exc:`OSError` instead of :exc:`NotImplementedError`. - - .. method:: set_data(path, data) - - Optional abstract method which writes the specified bytes to a file - path. Any intermediate directories which do not exist are to be created - automatically. - - When writing to the path fails because the path is read-only - (:attr:`errno.EACCES`/:exc:`PermissionError`), do not propagate the - exception. - - .. versionchanged:: 3.4 - No longer raises :exc:`NotImplementedError` when called. - - .. method:: get_code(fullname) - - Concrete implementation of :meth:`InspectLoader.get_code`. - - .. method:: exec_module(module) - - Concrete implementation of :meth:`Loader.exec_module`. - - .. versionadded:: 3.4 - - .. method:: load_module(fullname) - - Concrete implementation of :meth:`Loader.load_module`. - - .. deprecated:: 3.4 - Use :meth:`exec_module` instead. - - .. method:: get_source(fullname) - - Concrete implementation of :meth:`InspectLoader.get_source`. - - .. method:: is_package(fullname) - - Concrete implementation of :meth:`InspectLoader.is_package`. A module - is determined to be a package if its file path (as provided by - :meth:`ExecutionLoader.get_filename`) is a file named - ``__init__`` when the file extension is removed **and** the module name - itself does not end in ``__init__``. - - -.. class:: Traversable - - An object with a subset of pathlib.Path methods suitable for - traversing directories and opening files. - - .. versionadded:: 3.9 - - .. attribute:: name - - Abstract. The base name of this object without any parent references. - - .. abstractmethod:: iterdir() - - Yield Traversable objects in self. - - .. abstractmethod:: is_dir() - - Return True if self is a directory. - - .. abstractmethod:: is_file() - - Return True if self is a file. - - .. abstractmethod:: joinpath(child) - - Return Traversable child in self. - - .. abstractmethod:: __truediv__(child) - - Return Traversable child in self. - - .. abstractmethod:: open(mode='r', *args, **kwargs) - - *mode* may be 'r' or 'rb' to open as text or binary. Return a handle - suitable for reading (same as :attr:`pathlib.Path.open`). - - When opening as text, accepts encoding parameters such as those - accepted by :attr:`io.TextIOWrapper`. - - .. method:: read_bytes() - - Read contents of self as bytes. - - .. method:: read_text(encoding=None) - - Read contents of self as text. - - -.. class:: TraversableResources - - An abstract base class for resource readers capable of serving - the :meth:`importlib.resources.files` interface. Subclasses - :class:`importlib.abc.ResourceReader` and provides - concrete implementations of the :class:`importlib.abc.ResourceReader`'s - abstract methods. Therefore, any loader supplying - :class:`importlib.abc.TraversableReader` also supplies ResourceReader. - - Loaders that wish to support resource reading are expected to - implement this interface. - - .. versionadded:: 3.9 - - .. abstractmethod:: files() - - Returns a :class:`importlib.abc.Traversable` object for the loaded - package. - - -:mod:`importlib.resources` -- Resources ---------------------------------------- - -.. module:: importlib.resources - :synopsis: Package resource reading, opening, and access - -**Source code:** :source:`Lib/importlib/resources.py` - --------------- - -.. versionadded:: 3.7 - -This module leverages Python's import system to provide access to *resources* -within *packages*. If you can import a package, you can access resources -within that package. Resources can be opened or read, in either binary or -text mode. - -Resources are roughly akin to files inside directories, though it's important -to keep in mind that this is just a metaphor. Resources and packages **do -not** have to exist as physical files and directories on the file system. - -.. note:: - - This module provides functionality similar to `pkg_resources - <https://setuptools.readthedocs.io/en/latest/pkg_resources.html>`_ `Basic - Resource Access - <http://setuptools.readthedocs.io/en/latest/pkg_resources.html#basic-resourc…>`_ - without the performance overhead of that package. This makes reading - resources included in packages easier, with more stable and consistent - semantics. - - The standalone backport of this module provides more information - on `using importlib.resources - <http://importlib-resources.readthedocs.io/en/latest/using.html>`_ and - `migrating from pkg_resources to importlib.resources - <http://importlib-resources.readthedocs.io/en/latest/migration.html>`_ - and - `migrating legacy usage <https://importlib-resources.readthedocs.io/en/latest/using.html#migrating-f…>`_. - -Loaders that wish to support resource reading should implement a -``get_resource_reader(fullname)`` method as specified by -:class:`importlib.abc.ResourceReader`. - -The following types are defined. - -.. data:: Package - - The ``Package`` type is defined as ``Union[str, ModuleType]``. This means - that where the function describes accepting a ``Package``, you can pass in - either a string or a module. Module objects must have a resolvable - ``__spec__.submodule_search_locations`` that is not ``None``. - -.. data:: Resource - - This type describes the resource names passed into the various functions - in this package. This is defined as ``Union[str, os.PathLike]``. - - -The following functions are available. - - -.. function:: files(package) - - Returns an :class:`importlib.resources.abc.Traversable` object - representing the resource container for the package (think directory) - and its resources (think files). A Traversable may contain other - containers (think subdirectories). - - *package* is either a name or a module object which conforms to the - ``Package`` requirements. - - .. versionadded:: 3.9 - -.. function:: as_file(traversable) - - Given a :class:`importlib.resources.abc.Traversable` object representing - a file, typically from :func:`importlib.resources.files`, return - a context manager for use in a :keyword:`with` statement. - The context manager provides a :class:`pathlib.Path` object. - - Exiting the context manager cleans up any temporary file created when the - resource was extracted from e.g. a zip file. - - Use ``as_file`` when the Traversable methods - (``read_text``, etc) are insufficient and an actual file on - the file system is required. - - .. versionadded:: 3.9 - -.. function:: open_binary(package, resource) - - Open for binary reading the *resource* within *package*. - - *package* is either a name or a module object which conforms to the - ``Package`` requirements. *resource* is the name of the resource to open - within *package*; it may not contain path separators and it may not have - sub-resources (i.e. it cannot be a directory). This function returns a - ``typing.BinaryIO`` instance, a binary I/O stream open for reading. - - .. deprecated:: 3.11 - - -.. function:: open_text(package, resource, encoding='utf-8', errors='strict') - - Open for text reading the *resource* within *package*. By default, the - resource is opened for reading as UTF-8. - - *package* is either a name or a module object which conforms to the - ``Package`` requirements. *resource* is the name of the resource to open - within *package*; it may not contain path separators and it may not have - sub-resources (i.e. it cannot be a directory). *encoding* and *errors* - have the same meaning as with built-in :func:`open`. - - This function returns a ``typing.TextIO`` instance, a text I/O stream open - for reading. - - .. deprecated:: 3.11 - - -.. function:: read_binary(package, resource) - - Read and return the contents of the *resource* within *package* as - ``bytes``. - - *package* is either a name or a module object which conforms to the - ``Package`` requirements. *resource* is the name of the resource to open - within *package*; it may not contain path separators and it may not have - sub-resources (i.e. it cannot be a directory). This function returns the - contents of the resource as :class:`bytes`. - - .. deprecated:: 3.11 - - -.. function:: read_text(package, resource, encoding='utf-8', errors='strict') - - Read and return the contents of *resource* within *package* as a ``str``. - By default, the contents are read as strict UTF-8. - - *package* is either a name or a module object which conforms to the - ``Package`` requirements. *resource* is the name of the resource to open - within *package*; it may not contain path separators and it may not have - sub-resources (i.e. it cannot be a directory). *encoding* and *errors* - have the same meaning as with built-in :func:`open`. This function - returns the contents of the resource as :class:`str`. - - .. deprecated:: 3.11 - - -.. function:: path(package, resource) - - Return the path to the *resource* as an actual file system path. This - function returns a context manager for use in a :keyword:`with` statement. - The context manager provides a :class:`pathlib.Path` object. - - Exiting the context manager cleans up any temporary file created when the - resource needs to be extracted from e.g. a zip file. - - *package* is either a name or a module object which conforms to the - ``Package`` requirements. *resource* is the name of the resource to open - within *package*; it may not contain path separators and it may not have - sub-resources (i.e. it cannot be a directory). - - .. deprecated:: 3.11 - - -.. function:: is_resource(package, name) - - Return ``True`` if there is a resource named *name* in the package, - otherwise ``False``. Remember that directories are *not* resources! - *package* is either a name or a module object which conforms to the - ``Package`` requirements. - - .. deprecated:: 3.11 - - -.. function:: contents(package) - - Return an iterable over the named items within the package. The iterable - returns :class:`str` resources (e.g. files) and non-resources - (e.g. directories). The iterable does not recurse into subdirectories. - - *package* is either a name or a module object which conforms to the - ``Package`` requirements. - - .. deprecated:: 3.11 +.. include:: importlib.resources.abc.rst :mod:`importlib.machinery` -- Importers and path hooks diff --git a/Doc/library/modules.rst b/Doc/library/modules.rst index 565ce0525c2c9..131dfca9576a3 100644 --- a/Doc/library/modules.rst +++ b/Doc/library/modules.rst @@ -17,4 +17,5 @@ The full list of modules described in this chapter is: modulefinder.rst runpy.rst importlib.rst + importlib.resources.rst importlib.metadata.rst diff --git a/Misc/NEWS.d/next/Documentation/2021-12-16-21-13-55.bpo-46109.0-RNzu.rst b/Misc/NEWS.d/next/Documentation/2021-12-16-21-13-55.bpo-46109.0-RNzu.rst new file mode 100644 index 0000000000000..78d5149c80022 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2021-12-16-21-13-55.bpo-46109.0-RNzu.rst @@ -0,0 +1,2 @@ +Extracted ``importlib.resources`` and ``importlib.resources.abc`` documentation into +separate files.

1 0

bpo-46118: Move importlib.resources to its own package. (#30176)
by jaraco 31 Dec '21

31 Dec '21

https://github.com/python/cpython/commit/e712a5b277866a71c195f38c1b5d87d912… commit: e712a5b277866a71c195f38c1b5d87d9126dba3e branch: main author: Jason R. Coombs <jaraco(a)jaraco.com> committer: jaraco <jaraco(a)jaraco.com> date: 2021-12-30T21:00:48-05:00 summary: bpo-46118: Move importlib.resources to its own package. (#30176) * bpo-46118: Move importlib.resources to its own package. * Expand compatibility shims with documentation and explicit imports. files: A Lib/importlib/resources/__init__.py A Lib/importlib/resources/_adapters.py A Lib/importlib/resources/_common.py A Lib/importlib/resources/_itertools.py A Lib/importlib/resources/_legacy.py A Lib/importlib/resources/abc.py A Lib/importlib/resources/readers.py A Lib/importlib/resources/simple.py A Misc/NEWS.d/next/Library/2021-12-17-16-27-44.bpo-46118.euAy0E.rst D Lib/importlib/_adapters.py D Lib/importlib/_common.py D Lib/importlib/_itertools.py D Lib/importlib/_legacy.py D Lib/importlib/resources.py M Lib/importlib/abc.py M Lib/importlib/readers.py M Lib/importlib/simple.py M Lib/test/test_importlib/test_compatibilty_files.py diff --git a/Lib/importlib/abc.py b/Lib/importlib/abc.py index f10441608ad71..3fa151f390ba7 100644 --- a/Lib/importlib/abc.py +++ b/Lib/importlib/abc.py @@ -14,8 +14,19 @@ from ._abc import Loader import abc import warnings -from typing import BinaryIO, Iterable, Text -from typing import Protocol, runtime_checkable + +# for compatibility with Python 3.10 +from .resources.abc import ResourceReader, Traversable, TraversableResources + + +__all__ = [ + 'Loader', 'Finder', 'MetaPathFinder', 'PathEntryFinder', + 'ResourceLoader', 'InspectLoader', 'ExecutionLoader', + 'FileLoader', 'SourceLoader', + + # for compatibility with Python 3.10 + 'ResourceReader', 'Traversable', 'TraversableResources', +] def _register(abstract_cls, *classes): @@ -307,136 +318,3 @@ def set_data(self, path, data): """ _register(SourceLoader, machinery.SourceFileLoader) - - -class ResourceReader(metaclass=abc.ABCMeta): - """Abstract base class for loaders to provide resource reading support.""" - - @abc.abstractmethod - def open_resource(self, resource: Text) -> BinaryIO: - """Return an opened, file-like object for binary reading. - - The 'resource' argument is expected to represent only a file name. - If the resource cannot be found, FileNotFoundError is raised. - """ - # This deliberately raises FileNotFoundError instead of - # NotImplementedError so that if this method is accidentally called, - # it'll still do the right thing. - raise FileNotFoundError - - @abc.abstractmethod - def resource_path(self, resource: Text) -> Text: - """Return the file system path to the specified resource. - - The 'resource' argument is expected to represent only a file name. - If the resource does not exist on the file system, raise - FileNotFoundError. - """ - # This deliberately raises FileNotFoundError instead of - # NotImplementedError so that if this method is accidentally called, - # it'll still do the right thing. - raise FileNotFoundError - - @abc.abstractmethod - def is_resource(self, path: Text) -> bool: - """Return True if the named 'path' is a resource. - - Files are resources, directories are not. - """ - raise FileNotFoundError - - @abc.abstractmethod - def contents(self) -> Iterable[str]: - """Return an iterable of entries in `package`.""" - raise FileNotFoundError - - -@runtime_checkable -class Traversable(Protocol): - """ - An object with a subset of pathlib.Path methods suitable for - traversing directories and opening files. - """ - - @abc.abstractmethod - def iterdir(self): - """ - Yield Traversable objects in self - """ - - def read_bytes(self): - """ - Read contents of self as bytes - """ - with self.open('rb') as strm: - return strm.read() - - def read_text(self, encoding=None): - """ - Read contents of self as text - """ - with self.open(encoding=encoding) as strm: - return strm.read() - - @abc.abstractmethod - def is_dir(self) -> bool: - """ - Return True if self is a directory - """ - - @abc.abstractmethod - def is_file(self) -> bool: - """ - Return True if self is a file - """ - - @abc.abstractmethod - def joinpath(self, child): - """ - Return Traversable child in self - """ - - def __truediv__(self, child): - """ - Return Traversable child in self - """ - return self.joinpath(child) - - @abc.abstractmethod - def open(self, mode='r', *args, **kwargs): - """ - mode may be 'r' or 'rb' to open as text or binary. Return a handle - suitable for reading (same as pathlib.Path.open). - - When opening as text, accepts encoding parameters such as those - accepted by io.TextIOWrapper. - """ - - @abc.abstractproperty - def name(self) -> str: - """ - The base name of this object without any parent references. - """ - - -class TraversableResources(ResourceReader): - """ - The required interface for providing traversable - resources. - """ - - @abc.abstractmethod - def files(self): - """Return a Traversable object for the loaded package.""" - - def open_resource(self, resource): - return self.files().joinpath(resource).open('rb') - - def resource_path(self, resource): - raise FileNotFoundError(resource) - - def is_resource(self, path): - return self.files().joinpath(path).is_file() - - def contents(self): - return (item.name for item in self.files().iterdir()) diff --git a/Lib/importlib/readers.py b/Lib/importlib/readers.py index b470a2062b2b3..df7fb92e5cd9f 100644 --- a/Lib/importlib/readers.py +++ b/Lib/importlib/readers.py @@ -1,122 +1,12 @@ -import collections -import operator -import pathlib -import zipfile +""" +Compatibility shim for .resources.readers as found on Python 3.10. -from . import abc +Consumers that can rely on Python 3.11 should use the other +module directly. +""" -from ._itertools import unique_everseen +from .resources.readers import ( + FileReader, ZipReader, MultiplexedPath, NamespaceReader, +) - -def remove_duplicates(items): - return iter(collections.OrderedDict.fromkeys(items)) - - -class FileReader(abc.TraversableResources): - def __init__(self, loader): - self.path = pathlib.Path(loader.path).parent - - def resource_path(self, resource): - """ - Return the file system path to prevent - `resources.path()` from creating a temporary - copy. - """ - return str(self.path.joinpath(resource)) - - def files(self): - return self.path - - -class ZipReader(abc.TraversableResources): - def __init__(self, loader, module): - _, _, name = module.rpartition('.') - self.prefix = loader.prefix.replace('\\', '/') + name + '/' - self.archive = loader.archive - - def open_resource(self, resource): - try: - return super().open_resource(resource) - except KeyError as exc: - raise FileNotFoundError(exc.args[0]) - - def is_resource(self, path): - # workaround for `zipfile.Path.is_file` returning true - # for non-existent paths. - target = self.files().joinpath(path) - return target.is_file() and target.exists() - - def files(self): - return zipfile.Path(self.archive, self.prefix) - - -class MultiplexedPath(abc.Traversable): - """ - Given a series of Traversable objects, implement a merged - version of the interface across all objects. Useful for - namespace packages which may be multihomed at a single - name. - """ - - def __init__(self, *paths): - self._paths = list(map(pathlib.Path, remove_duplicates(paths))) - if not self._paths: - message = 'MultiplexedPath must contain at least one path' - raise FileNotFoundError(message) - if not all(path.is_dir() for path in self._paths): - raise NotADirectoryError('MultiplexedPath only supports directories') - - def iterdir(self): - files = (file for path in self._paths for file in path.iterdir()) - return unique_everseen(files, key=operator.attrgetter('name')) - - def read_bytes(self): - raise FileNotFoundError(f'{self} is not a file') - - def read_text(self, *args, **kwargs): - raise FileNotFoundError(f'{self} is not a file') - - def is_dir(self): - return True - - def is_file(self): - return False - - def joinpath(self, child): - # first try to find child in current paths - for file in self.iterdir(): - if file.name == child: - return file - # if it does not exist, construct it with the first path - return self._paths[0] / child - - __truediv__ = joinpath - - def open(self, *args, **kwargs): - raise FileNotFoundError(f'{self} is not a file') - - @property - def name(self): - return self._paths[0].name - - def __repr__(self): - paths = ', '.join(f"'{path}'" for path in self._paths) - return f'MultiplexedPath({paths})' - - -class NamespaceReader(abc.TraversableResources): - def __init__(self, namespace_path): - if 'NamespacePath' not in str(namespace_path): - raise ValueError('Invalid path') - self.path = MultiplexedPath(*list(namespace_path)) - - def resource_path(self, resource): - """ - Return the file system path to prevent - `resources.path()` from creating a temporary - copy. - """ - return str(self.path.joinpath(resource)) - - def files(self): - return self.path +__all__ = ['FileReader', 'ZipReader', 'MultiplexedPath', 'NamespaceReader'] diff --git a/Lib/importlib/resources.py b/Lib/importlib/resources/__init__.py similarity index 92% rename from Lib/importlib/resources.py rename to Lib/importlib/resources/__init__.py index 5b3bc0228d9ac..34e3a9950cc55 100644 --- a/Lib/importlib/resources.py +++ b/Lib/importlib/resources/__init__.py @@ -17,7 +17,7 @@ Resource, ) -from importlib.abc import ResourceReader +from .abc import ResourceReader __all__ = [ diff --git a/Lib/importlib/_adapters.py b/Lib/importlib/resources/_adapters.py similarity index 100% rename from Lib/importlib/_adapters.py rename to Lib/importlib/resources/_adapters.py diff --git a/Lib/importlib/_common.py b/Lib/importlib/resources/_common.py similarity index 100% rename from Lib/importlib/_common.py rename to Lib/importlib/resources/_common.py diff --git a/Lib/importlib/_itertools.py b/Lib/importlib/resources/_itertools.py similarity index 100% rename from Lib/importlib/_itertools.py rename to Lib/importlib/resources/_itertools.py diff --git a/Lib/importlib/_legacy.py b/Lib/importlib/resources/_legacy.py similarity index 100% rename from Lib/importlib/_legacy.py rename to Lib/importlib/resources/_legacy.py diff --git a/Lib/importlib/resources/abc.py b/Lib/importlib/resources/abc.py new file mode 100644 index 0000000000000..e9efdab5ea830 --- /dev/null +++ b/Lib/importlib/resources/abc.py @@ -0,0 +1,136 @@ +import abc +from typing import BinaryIO, Iterable, Text +from typing import runtime_checkable, Protocol + + +class ResourceReader(metaclass=abc.ABCMeta): + """Abstract base class for loaders to provide resource reading support.""" + + @abc.abstractmethod + def open_resource(self, resource: Text) -> BinaryIO: + """Return an opened, file-like object for binary reading. + + The 'resource' argument is expected to represent only a file name. + If the resource cannot be found, FileNotFoundError is raised. + """ + # This deliberately raises FileNotFoundError instead of + # NotImplementedError so that if this method is accidentally called, + # it'll still do the right thing. + raise FileNotFoundError + + @abc.abstractmethod + def resource_path(self, resource: Text) -> Text: + """Return the file system path to the specified resource. + + The 'resource' argument is expected to represent only a file name. + If the resource does not exist on the file system, raise + FileNotFoundError. + """ + # This deliberately raises FileNotFoundError instead of + # NotImplementedError so that if this method is accidentally called, + # it'll still do the right thing. + raise FileNotFoundError + + @abc.abstractmethod + def is_resource(self, path: Text) -> bool: + """Return True if the named 'path' is a resource. + + Files are resources, directories are not. + """ + raise FileNotFoundError + + @abc.abstractmethod + def contents(self) -> Iterable[str]: + """Return an iterable of entries in `package`.""" + raise FileNotFoundError + + +@runtime_checkable +class Traversable(Protocol): + """ + An object with a subset of pathlib.Path methods suitable for + traversing directories and opening files. + """ + + @abc.abstractmethod + def iterdir(self): + """ + Yield Traversable objects in self + """ + + def read_bytes(self): + """ + Read contents of self as bytes + """ + with self.open('rb') as strm: + return strm.read() + + def read_text(self, encoding=None): + """ + Read contents of self as text + """ + with self.open(encoding=encoding) as strm: + return strm.read() + + @abc.abstractmethod + def is_dir(self) -> bool: + """ + Return True if self is a directory + """ + + @abc.abstractmethod + def is_file(self) -> bool: + """ + Return True if self is a file + """ + + @abc.abstractmethod + def joinpath(self, child): + """ + Return Traversable child in self + """ + + def __truediv__(self, child): + """ + Return Traversable child in self + """ + return self.joinpath(child) + + @abc.abstractmethod + def open(self, mode='r', *args, **kwargs): + """ + mode may be 'r' or 'rb' to open as text or binary. Return a handle + suitable for reading (same as pathlib.Path.open). + + When opening as text, accepts encoding parameters such as those + accepted by io.TextIOWrapper. + """ + + @abc.abstractproperty + def name(self) -> str: + """ + The base name of this object without any parent references. + """ + + +class TraversableResources(ResourceReader): + """ + The required interface for providing traversable + resources. + """ + + @abc.abstractmethod + def files(self): + """Return a Traversable object for the loaded package.""" + + def open_resource(self, resource): + return self.files().joinpath(resource).open('rb') + + def resource_path(self, resource): + raise FileNotFoundError(resource) + + def is_resource(self, path): + return self.files().joinpath(path).is_file() + + def contents(self): + return (item.name for item in self.files().iterdir()) diff --git a/Lib/importlib/resources/readers.py b/Lib/importlib/resources/readers.py new file mode 100644 index 0000000000000..b470a2062b2b3 --- /dev/null +++ b/Lib/importlib/resources/readers.py @@ -0,0 +1,122 @@ +import collections +import operator +import pathlib +import zipfile + +from . import abc + +from ._itertools import unique_everseen + + +def remove_duplicates(items): + return iter(collections.OrderedDict.fromkeys(items)) + + +class FileReader(abc.TraversableResources): + def __init__(self, loader): + self.path = pathlib.Path(loader.path).parent + + def resource_path(self, resource): + """ + Return the file system path to prevent + `resources.path()` from creating a temporary + copy. + """ + return str(self.path.joinpath(resource)) + + def files(self): + return self.path + + +class ZipReader(abc.TraversableResources): + def __init__(self, loader, module): + _, _, name = module.rpartition('.') + self.prefix = loader.prefix.replace('\\', '/') + name + '/' + self.archive = loader.archive + + def open_resource(self, resource): + try: + return super().open_resource(resource) + except KeyError as exc: + raise FileNotFoundError(exc.args[0]) + + def is_resource(self, path): + # workaround for `zipfile.Path.is_file` returning true + # for non-existent paths. + target = self.files().joinpath(path) + return target.is_file() and target.exists() + + def files(self): + return zipfile.Path(self.archive, self.prefix) + + +class MultiplexedPath(abc.Traversable): + """ + Given a series of Traversable objects, implement a merged + version of the interface across all objects. Useful for + namespace packages which may be multihomed at a single + name. + """ + + def __init__(self, *paths): + self._paths = list(map(pathlib.Path, remove_duplicates(paths))) + if not self._paths: + message = 'MultiplexedPath must contain at least one path' + raise FileNotFoundError(message) + if not all(path.is_dir() for path in self._paths): + raise NotADirectoryError('MultiplexedPath only supports directories') + + def iterdir(self): + files = (file for path in self._paths for file in path.iterdir()) + return unique_everseen(files, key=operator.attrgetter('name')) + + def read_bytes(self): + raise FileNotFoundError(f'{self} is not a file') + + def read_text(self, *args, **kwargs): + raise FileNotFoundError(f'{self} is not a file') + + def is_dir(self): + return True + + def is_file(self): + return False + + def joinpath(self, child): + # first try to find child in current paths + for file in self.iterdir(): + if file.name == child: + return file + # if it does not exist, construct it with the first path + return self._paths[0] / child + + __truediv__ = joinpath + + def open(self, *args, **kwargs): + raise FileNotFoundError(f'{self} is not a file') + + @property + def name(self): + return self._paths[0].name + + def __repr__(self): + paths = ', '.join(f"'{path}'" for path in self._paths) + return f'MultiplexedPath({paths})' + + +class NamespaceReader(abc.TraversableResources): + def __init__(self, namespace_path): + if 'NamespacePath' not in str(namespace_path): + raise ValueError('Invalid path') + self.path = MultiplexedPath(*list(namespace_path)) + + def resource_path(self, resource): + """ + Return the file system path to prevent + `resources.path()` from creating a temporary + copy. + """ + return str(self.path.joinpath(resource)) + + def files(self): + return self.path diff --git a/Lib/importlib/resources/simple.py b/Lib/importlib/resources/simple.py new file mode 100644 index 0000000000000..da073cbdb11e6 --- /dev/null +++ b/Lib/importlib/resources/simple.py @@ -0,0 +1,116 @@ +""" +Interface adapters for low-level readers. +""" + +import abc +import io +import itertools +from typing import BinaryIO, List + +from .abc import Traversable, TraversableResources + + +class SimpleReader(abc.ABC): + """ + The minimum, low-level interface required from a resource + provider. + """ + + @abc.abstractproperty + def package(self): + # type: () -> str + """ + The name of the package for which this reader loads resources. + """ + + @abc.abstractmethod + def children(self): + # type: () -> List['SimpleReader'] + """ + Obtain an iterable of SimpleReader for available + child containers (e.g. directories). + """ + + @abc.abstractmethod + def resources(self): + # type: () -> List[str] + """ + Obtain available named resources for this virtual package. + """ + + @abc.abstractmethod + def open_binary(self, resource): + # type: (str) -> BinaryIO + """ + Obtain a File-like for a named resource. + """ + + @property + def name(self): + return self.package.split('.')[-1] + + +class ResourceHandle(Traversable): + """ + Handle to a named resource in a ResourceReader. + """ + + def __init__(self, parent, name): + # type: (ResourceContainer, str) -> None + self.parent = parent + self.name = name # type: ignore + + def is_file(self): + return True + + def is_dir(self): + return False + + def open(self, mode='r', *args, **kwargs): + stream = self.parent.reader.open_binary(self.name) + if 'b' not in mode: + stream = io.TextIOWrapper(*args, **kwargs) + return stream + + def joinpath(self, name): + raise RuntimeError("Cannot traverse into a resource") + + +class ResourceContainer(Traversable): + """ + Traversable container for a package's resources via its reader. + """ + + def __init__(self, reader): + # type: (SimpleReader) -> None + self.reader = reader + + def is_dir(self): + return True + + def is_file(self): + return False + + def iterdir(self): + files = (ResourceHandle(self, name) for name in self.reader.resources) + dirs = map(ResourceContainer, self.reader.children()) + return itertools.chain(files, dirs) + + def open(self, *args, **kwargs): + raise IsADirectoryError() + + def joinpath(self, name): + return next( + traversable for traversable in self.iterdir() if traversable.name == name + ) + + +class TraversableReader(TraversableResources, SimpleReader): + """ + A TraversableResources based on SimpleReader. Resource providers + may derive from this class to provide the TraversableResources + interface by supplying the SimpleReader interface. + """ + + def files(self): + return ResourceContainer(self) diff --git a/Lib/importlib/simple.py b/Lib/importlib/simple.py index da073cbdb11e6..845bb90364784 100644 --- a/Lib/importlib/simple.py +++ b/Lib/importlib/simple.py @@ -1,116 +1,14 @@ """ -Interface adapters for low-level readers. -""" - -import abc -import io -import itertools -from typing import BinaryIO, List - -from .abc import Traversable, TraversableResources - - -class SimpleReader(abc.ABC): - """ - The minimum, low-level interface required from a resource - provider. - """ - - @abc.abstractproperty - def package(self): - # type: () -> str - """ - The name of the package for which this reader loads resources. - """ - - @abc.abstractmethod - def children(self): - # type: () -> List['SimpleReader'] - """ - Obtain an iterable of SimpleReader for available - child containers (e.g. directories). - """ - - @abc.abstractmethod - def resources(self): - # type: () -> List[str] - """ - Obtain available named resources for this virtual package. - """ - - @abc.abstractmethod - def open_binary(self, resource): - # type: (str) -> BinaryIO - """ - Obtain a File-like for a named resource. - """ - - @property - def name(self): - return self.package.split('.')[-1] - - -class ResourceHandle(Traversable): - """ - Handle to a named resource in a ResourceReader. - """ - - def __init__(self, parent, name): - # type: (ResourceContainer, str) -> None - self.parent = parent - self.name = name # type: ignore - - def is_file(self): - return True - - def is_dir(self): - return False - - def open(self, mode='r', *args, **kwargs): - stream = self.parent.reader.open_binary(self.name) - if 'b' not in mode: - stream = io.TextIOWrapper(*args, **kwargs) - return stream - - def joinpath(self, name): - raise RuntimeError("Cannot traverse into a resource") - - -class ResourceContainer(Traversable): - """ - Traversable container for a package's resources via its reader. - """ - - def __init__(self, reader): - # type: (SimpleReader) -> None - self.reader = reader - - def is_dir(self): - return True - - def is_file(self): - return False - - def iterdir(self): - files = (ResourceHandle(self, name) for name in self.reader.resources) - dirs = map(ResourceContainer, self.reader.children()) - return itertools.chain(files, dirs) - - def open(self, *args, **kwargs): - raise IsADirectoryError() - - def joinpath(self, name): - return next( - traversable for traversable in self.iterdir() if traversable.name == name - ) +Compatibility shim for .resources.simple as found on Python 3.10. +Consumers that can rely on Python 3.11 should use the other +module directly. +""" -class TraversableReader(TraversableResources, SimpleReader): - """ - A TraversableResources based on SimpleReader. Resource providers - may derive from this class to provide the TraversableResources - interface by supplying the SimpleReader interface. - """ +from .resources.simple import ( + SimpleReader, ResourceHandle, ResourceContainer, TraversableReader, +) - def files(self): - return ResourceContainer(self) +__all__ = [ + 'SimpleReader', 'ResourceHandle', 'ResourceContainer', 'TraversableReader', +] diff --git a/Lib/test/test_importlib/test_compatibilty_files.py b/Lib/test/test_importlib/test_compatibilty_files.py index d703c060c4407..9a823f2d93058 100644 --- a/Lib/test/test_importlib/test_compatibilty_files.py +++ b/Lib/test/test_importlib/test_compatibilty_files.py @@ -3,7 +3,7 @@ from importlib import resources -from importlib._adapters import ( +from importlib.resources._adapters import ( CompatibilityFiles, wrap_spec, ) diff --git a/Misc/NEWS.d/next/Library/2021-12-17-16-27-44.bpo-46118.euAy0E.rst b/Misc/NEWS.d/next/Library/2021-12-17-16-27-44.bpo-46118.euAy0E.rst new file mode 100644 index 0000000000000..c53e5765b9785 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-12-17-16-27-44.bpo-46118.euAy0E.rst @@ -0,0 +1 @@ +Moved importlib.resources and its related functionality to a package.

1 0

bpo-46178: Remove/rename redundant Travis CI code (#30309)
by Mariatta 31 Dec '21

31 Dec '21

https://github.com/python/cpython/commit/2cf7d02b99ce8cebd26d330aa8aac2ee36… commit: 2cf7d02b99ce8cebd26d330aa8aac2ee369d4600 branch: main author: Hugo van Kemenade <hugovk(a)users.noreply.github.com> committer: Mariatta <Mariatta(a)users.noreply.github.com> date: 2021-12-30T16:16:27-08:00 summary: bpo-46178: Remove/rename redundant Travis CI code (#30309) files: M .azure-pipelines/posix-steps.yml M Doc/library/unittest.rst M Lib/test/test_urllib2net.py M README.rst M Tools/scripts/patchcheck.py diff --git a/.azure-pipelines/posix-steps.yml b/.azure-pipelines/posix-steps.yml index b6dde593019e6..2a3680897302e 100644 --- a/.azure-pipelines/posix-steps.yml +++ b/.azure-pipelines/posix-steps.yml @@ -66,7 +66,7 @@ steps: COMMAND: make - ${{ if eq(parameters.patchcheck, 'true') }}: - - script: ./python Tools/scripts/patchcheck.py --travis true + - script: ./python Tools/scripts/patchcheck.py --ci true displayName: 'Run patchcheck.py' condition: and(succeeded(), eq(variables['Build.Reason'], 'PullRequest')) diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst index bf7a1e87928a6..22723f42d048f 100644 --- a/Doc/library/unittest.rst +++ b/Doc/library/unittest.rst @@ -72,8 +72,9 @@ test runner a GUI tool for test discovery and execution. This is intended largely for ease of use for those new to unit testing. For production environments it is recommended that tests be driven by a continuous integration system such as - `Buildbot <https://buildbot.net/>`_, `Jenkins <https://jenkins.io/>`_ - or `Travis-CI <https://travis-ci.com>`_, or `AppVeyor <https://www.appveyor.com/>`_. + `Buildbot <https://buildbot.net/>`_, `Jenkins <https://jenkins.io/>`_, + `GitHub Actions <https://github.com/features/actions>`_, or + `AppVeyor <https://www.appveyor.com/>`_. .. _unittest-minimal-example: diff --git a/Lib/test/test_urllib2net.py b/Lib/test/test_urllib2net.py index aa41811560c78..04cfb492e4549 100644 --- a/Lib/test/test_urllib2net.py +++ b/Lib/test/test_urllib2net.py @@ -28,13 +28,6 @@ def wrapped(*args, **kwargs): return _retry_thrice(func, exc, *args, **kwargs) return wrapped -# bpo-35411: FTP tests of test_urllib2net randomly fail -# with "425 Security: Bad IP connecting" on Travis CI -skip_ftp_test_on_travis = unittest.skipIf('TRAVIS' in os.environ, - 'bpo-35411: skip FTP test ' - 'on Travis CI') - - # Connecting to remote hosts is flaky. Make it more robust by retrying # the connection several times. _urlopen_with_retry = _wrap_with_retry_thrice(urllib.request.urlopen, @@ -139,7 +132,6 @@ def setUp(self): # XXX The rest of these tests aren't very good -- they don't check much. # They do sometimes catch some major disasters, though. - @skip_ftp_test_on_travis def test_ftp(self): urls = [ 'ftp://www.pythontest.net/README', @@ -339,7 +331,6 @@ def test_http_timeout(self): FTP_HOST = 'ftp://www.pythontest.net/' - @skip_ftp_test_on_travis def test_ftp_basic(self): self.assertIsNone(socket.getdefaulttimeout()) with socket_helper.transient_internet(self.FTP_HOST, timeout=None): @@ -347,7 +338,6 @@ def test_ftp_basic(self): self.addCleanup(u.close) self.assertIsNone(u.fp.fp.raw._sock.gettimeout()) - @skip_ftp_test_on_travis def test_ftp_default_timeout(self): self.assertIsNone(socket.getdefaulttimeout()) with socket_helper.transient_internet(self.FTP_HOST): @@ -359,7 +349,6 @@ def test_ftp_default_timeout(self): socket.setdefaulttimeout(None) self.assertEqual(u.fp.fp.raw._sock.gettimeout(), 60) - @skip_ftp_test_on_travis def test_ftp_no_timeout(self): self.assertIsNone(socket.getdefaulttimeout()) with socket_helper.transient_internet(self.FTP_HOST): @@ -371,7 +360,6 @@ def test_ftp_no_timeout(self): socket.setdefaulttimeout(None) self.assertIsNone(u.fp.fp.raw._sock.gettimeout()) - @skip_ftp_test_on_travis def test_ftp_timeout(self): with socket_helper.transient_internet(self.FTP_HOST): u = _urlopen_with_retry(self.FTP_HOST, timeout=60) diff --git a/README.rst b/README.rst index 9f84432a55c97..d4b6621a80554 100644 --- a/README.rst +++ b/README.rst @@ -1,10 +1,6 @@ This is Python version 3.11.0 alpha 3 ===================================== -.. image:: https://travis-ci.com/python/cpython.svg?branch=main - :alt: CPython build status on Travis CI - :target: https://travis-ci.com/python/cpython - .. image:: https://github.com/python/cpython/workflows/Tests/badge.svg :alt: CPython build status on GitHub Actions :target: https://github.com/python/cpython/actions diff --git a/Tools/scripts/patchcheck.py b/Tools/scripts/patchcheck.py index 8e59c78a4c584..4cab66c8e97be 100755 --- a/Tools/scripts/patchcheck.py +++ b/Tools/scripts/patchcheck.py @@ -245,7 +245,7 @@ def regenerated_pyconfig_h_in(file_paths): else: return "not needed" -def travis(pull_request): +def ci(pull_request): if pull_request == 'false': print('Not a pull request; skipping') return @@ -301,10 +301,10 @@ def main(): if __name__ == '__main__': import argparse parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument('--travis', + parser.add_argument('--ci', help='Perform pass/fail checks') args = parser.parse_args() - if args.travis: - travis(args.travis) + if args.ci: + ci(args.ci) else: main()

1 0

bpo-46178: remove unusued `.travis.yml` file (#30257)
by Mariatta 30 Dec '21

30 Dec '21

https://github.com/python/cpython/commit/82c2b54a35f85d87639201d1431ae6584b… commit: 82c2b54a35f85d87639201d1431ae6584b079e13 branch: main author: Nikita Sobolev <mail(a)sobolevn.me> committer: Mariatta <Mariatta(a)users.noreply.github.com> date: 2021-12-30T13:26:01-08:00 summary: bpo-46178: remove unusued `.travis.yml` file (#30257) * bpo-46178: remove unusued `.travis.yml` file * Delete 2021-12-26-12-35-41.bpo-46178.Aw1TZg.rst files: D .travis.yml diff --git a/.travis.yml b/.travis.yml deleted file mode 100644 index b2f7e27458711..0000000000000 --- a/.travis.yml +++ /dev/null @@ -1,169 +0,0 @@ -language: c -dist: bionic - -# To cache doc-building dependencies and C compiler output. -cache: - - pip - - ccache - - directories: - - $HOME/multissl - -env: - global: - - OPENSSL=1.1.1k - - OPENSSL_DIR="$HOME/multissl/openssl/${OPENSSL}" - - PATH="${OPENSSL_DIR}/bin:$PATH" - - CFLAGS="-I${OPENSSL_DIR}/include" - - LDFLAGS="-L${OPENSSL_DIR}/lib" - # Set rpath with env var instead of -Wl,-rpath linker flag - # OpenSSL ignores LDFLAGS when linking bin/openssl - - LD_RUN_PATH="${OPENSSL_DIR}/lib" - - PYTHONSTRICTEXTENSIONBUILD=1 - -branches: - only: - - main - - /^\d\.\d+$/ - - buildbot-custom - -matrix: - fast_finish: true - allow_failures: - - env: OPTIONAL=true - include: - - name: "CPython tests" - os: linux - language: c - compiler: clang - # gcc also works, but to keep the # of concurrent builds down, we use one C - # compiler here and the other to run the coverage build. Clang is preferred - # in this instance for its better error messages. - env: TESTING=cpython - addons: - apt: - packages: - - gdb - - xvfb - - name: "Documentation build" - os: linux - language: python - # Build the docs against a stable version of Python so code bugs don't hold up doc-related PRs. - python: 3.6 - env: TESTING=docs - before_script: - - cd Doc - - make venv PYTHON=python - script: - - make check html SPHINXOPTS="-q -W -j4" - - name: "Documentation tests" - os: linux - language: c - compiler: clang - env: TESTING=doctest - addons: - apt: - packages: - - xvfb - before_script: - - ./configure - - make -j4 - - make -C Doc/ PYTHON=../python venv - script: - xvfb-run make -C Doc/ PYTHON=../python SPHINXOPTS="-q -W -j4" doctest - - -before_install: - - set -e - - | - # Check short-circuit conditions - if [[ "${TESTING}" != "docs" && "${TESTING}" != "doctest" ]] - then - if [[ "$TRAVIS_PULL_REQUEST" == "false" ]] - then - echo "Not a PR, doing full build." - else - # Pull requests are slightly complicated because $TRAVIS_COMMIT_RANGE - # may include more changes than desired if the history is convoluted. - # Instead, explicitly fetch the base branch and compare against the - # merge-base commit. - git fetch -q origin +refs/heads/$TRAVIS_BRANCH - changes=$(git diff --name-only HEAD $(git merge-base HEAD FETCH_HEAD)) - echo "Files changed:" - echo "$changes" - if ! echo "$changes" | grep -qvE '(\.rst$)|(^Doc)|(^Misc)' - then - echo "Only docs were updated, stopping build process." - exit - fi - fi - fi - -install: - - | - # Install OpenSSL as necessary - # Note: doctest needs OpenSSL - if [[ "${TESTING}" != "docs" ]] - then - # clang complains about unused-parameter a lot, redirect stderr - python3 Tools/ssl/multissltests.py --steps=library \ - --base-directory ${HOME}/multissl \ - --openssl ${OPENSSL} >/dev/null 2>&1 - fi - - openssl version - -# Travis provides only 2 cores, so don't overdo the parallelism and waste memory. -before_script: - # -Og is much faster than -O0 - - CFLAGS="${CFLAGS} -Og" ./configure --with-pydebug - - eval "$(pyenv init -)" - - pyenv global 3.8 - - PYTHON_FOR_REGEN=python3.8 make -j4 regen-all - - make regen-stdlib-module-names - - changes=`git status --porcelain` - - | - # Check for changes in regenerated files - if ! test -z "$changes" - then - echo "Generated files not up to date" - echo "$changes" - exit 1 - fi - - make -j4 - - make pythoninfo - -script: - # Using the built Python as patchcheck.py is built around the idea of using - # a checkout-build of CPython to know things like what base branch the changes - # should be compared against. - # Only run on Linux as the check only needs to be run once. - - if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then ./python Tools/scripts/patchcheck.py --travis $TRAVIS_PULL_REQUEST; fi - # Check that all symbols exported by libpython start with "Py" or "_Py" - - make smelly - # Check that all symbols in the limited abi are present - - make check-limited-abi - # `-r -w` implicitly provided through `make buildbottest`. - - | - if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then - XVFB_RUN=xvfb-run; - fi - $XVFB_RUN make buildbottest TESTOPTS="-j4 -uall,-cpu" -notifications: - email: false - irc: - channels: - # This is set to a secure variable to prevent forks from notifying the - # IRC channel whenever they fail a build. This can be removed when travis - # implements https://github.com/travis-ci/travis-ci/issues/1094. - # The actual value here is: irc.freenode.net#python-dev - - secure: "s7kAkpcom2yUJ8XqyjFI0obJmhAGrn1xmoivdaPdgBIA++X47TBp1x4pgDsbEsoalef7bEwa4l07KdT4qa+DOd/c4QxaWom7fbN3BuLVsZuVfODnl79+gYq/TAbGfyH+yDs18DXrUfPgwD7C5aW32ugsqAOd4iWzfGJQ5OrOZzqzGjYdYQUEkJFXgxDEIb4aHvxNDWGO3Po9uKISrhb5saQ0l776yLo1Ur7M4oxl8RTbCdgX0vf5TzPg52BgvZpOgt3DHOUYPeiJLKNjAE6ibg0U95sEvMfHX77nz4aFY4/3UI6FFaRla34rZ+mYKrn0TdxOhera1QOgPmM6HzdO4K44FpfK1DS0Xxk9U9/uApq+cG0bU3W+cVUHDBe5+90lpRBAXHeHCgT7TI8gec614aiT8lEr3+yH8OBRYGzkjNK8E2LJZ/SxnVxDe7aLF6AWcoWLfS6/ziAIBFQ5Nc4U72CT8fGVSkl8ywPiRlvixKdvTODMSZo0jMqlfZSNaAPTsNRx4wu5Uis4qekwe32Fz4aB6KGpsuuVjBi+H6v0RKxNJNGY3JKDiEH2TK0UE2auJ5GvLW48aUVFcQMB7euCWYXlSWVRHh3WLU8QXF29Dw4JduRZqUpOdRgMHU79UHRq+mkE0jAS/nBcS6CvsmxCpTSrfVYuMOu32yt18QQoTyU=" - on_success: change - on_failure: always - skip_join: true - webhooks: - urls: - # For the same reasons as above for IRC, we encrypt the webhook address - # for Zulip. The actual value is: - # https://python.zulipchat.com/api/v1/external/travis?api_key=<api-key-redacted>&stream=core%2Ftest+runs - - secure: "vLz2TodSL7wQ8DsIu86koRS9i4dsK40PH8+wzY93PBCCAzJAz113LTxK3/9PamMv+ObDRUSe5OpXcquE3d5Gwpu8IymF113qK0I3uNr+O3FtmKlj/Kd1P/V+z4pTpS3zh3lW9gyKV9EMWXIWS0IYKKZQU144XqUlIiucWK2jHJF/cSz2cRAx/6Kx68X4mZwEC7hiMOF4ZsWUMbCglM89ybeS0pr0kK9mmh88qsPyRQov3mRKswmVPlePk7clVLNAL43qSe3SzmrmACZfQ9KJYmpYnr/cjo2b6svYHcQBAwAUarZZ9KBMXeV7HwGWsSXAvHH2ynR0P++braBHGEMTGMSitdVWzFTmeiHnrkp08WAB+uFs54iEx3VklTk9bCzozTm2S94TRxbrsG9SypMvQxG570JV6P2XYuR+taCb/GMtMqrtGQm2e1Ht+nDLtiUb+/+rwEbicJJ13knptOQZI4tPOZESI/kXkORkSNwFfLSNLSl9jTlMmO7KjAAPApURHEdx26RbItAn8mIX2NcHTRjKn2qV4h3C54nmHmKWn/ZudHHJc6ieZSEUBoaLGAYmcWJRqrM6jiy2h9I9TRrCKAiGh5jT47FYKLwosTtV245l/ZhDb6eTVfEFT6TSLEoyfx9cCtTUvfMtXYl8eN9wlFYYpH8MSWbMD14eEkKBTWg=" - on_success: change - on_failure: always

1 0

bpo-46184: remove `netlify.toml` (#30272)
by Mariatta 30 Dec '21

30 Dec '21

https://github.com/python/cpython/commit/fbaf2e604cd354f1ebc6be029480010c67… commit: fbaf2e604cd354f1ebc6be029480010c6715a8ca branch: main author: Nikita Sobolev <mail(a)sobolevn.me> committer: Mariatta <Mariatta(a)users.noreply.github.com> date: 2021-12-30T13:18:37-08:00 summary: bpo-46184: remove `netlify.toml` (#30272) * bpo-46184: remove `netlify.toml` * Delete runtime.txt * Delete requirements.txt * Revert "Delete requirements.txt" This reverts commit 9aa4f0631f9d206ed7fddf37b43a24ec4e90fa7c. files: D Doc/runtime.txt D netlify.toml diff --git a/Doc/runtime.txt b/Doc/runtime.txt deleted file mode 100644 index 548d71365f0ec..0000000000000 --- a/Doc/runtime.txt +++ /dev/null @@ -1 +0,0 @@ -3.7 \ No newline at end of file diff --git a/netlify.toml b/netlify.toml deleted file mode 100644 index 387c8f954ada3..0000000000000 --- a/netlify.toml +++ /dev/null @@ -1,4 +0,0 @@ -[build] - base = "Doc/" - command = "make html" - publish = "Doc/build/html" \ No newline at end of file

1 0

Delete `FUNDING.yml`, since there's an organisation default (#30294)
by Mariatta 30 Dec '21

30 Dec '21

https://github.com/python/cpython/commit/30ee33d6bed37172f6ef97596bd98a5d78… commit: 30ee33d6bed37172f6ef97596bd98a5d78275ffb branch: main author: Nikita Sobolev <mail(a)sobolevn.me> committer: Mariatta <Mariatta(a)users.noreply.github.com> date: 2021-12-30T12:24:46-08:00 summary: Delete `FUNDING.yml`, since there's an organisation default (#30294) files: D .github/FUNDING.yml diff --git a/.github/FUNDING.yml b/.github/FUNDING.yml deleted file mode 100644 index b08459313accf..0000000000000 --- a/.github/FUNDING.yml +++ /dev/null @@ -1,2 +0,0 @@ -custom: https://www.python.org/psf/donations/python-dev/ -github: [python]

1 0

bpo-45853: Fix misspelling and unused import in pathlib (GH-30292)
by serhiy-storchaka 30 Dec '21

30 Dec '21

https://github.com/python/cpython/commit/8d7644fa64213207b8dc6f555cb8a02bfa… commit: 8d7644fa64213207b8dc6f555cb8a02bfabeced2 branch: main author: andrei kulakov <andrei.avk(a)gmail.com> committer: serhiy-storchaka <storchaka(a)gmail.com> date: 2021-12-30T09:45:06+02:00 summary: bpo-45853: Fix misspelling and unused import in pathlib (GH-30292) files: M Lib/pathlib.py diff --git a/Lib/pathlib.py b/Lib/pathlib.py index 621fba0e75c0f..f1a33178e2958 100644 --- a/Lib/pathlib.py +++ b/Lib/pathlib.py @@ -8,7 +8,7 @@ import sys import warnings from _collections_abc import Sequence -from errno import EINVAL, ENOENT, ENOTDIR, EBADF, ELOOP +from errno import ENOENT, ENOTDIR, EBADF, ELOOP from operator import attrgetter from stat import S_ISDIR, S_ISLNK, S_ISREG, S_ISSOCK, S_ISBLK, S_ISCHR, S_ISFIFO from urllib.parse import quote_from_bytes as urlquote_from_bytes @@ -28,7 +28,7 @@ _WINERROR_CANT_RESOLVE_FILENAME = 1921 # broken symlink pointing to itself # EBADF - guard against macOS `stat` throwing EBADF -_IGNORED_ERROS = (ENOENT, ENOTDIR, EBADF, ELOOP) +_IGNORED_ERRNOS = (ENOENT, ENOTDIR, EBADF, ELOOP) _IGNORED_WINERRORS = ( _WINERROR_NOT_READY, @@ -36,7 +36,7 @@ _WINERROR_CANT_RESOLVE_FILENAME) def _ignore_error(exception): - return (getattr(exception, 'errno', None) in _IGNORED_ERROS or + return (getattr(exception, 'errno', None) in _IGNORED_ERRNOS or getattr(exception, 'winerror', None) in _IGNORED_WINERRORS)

1 0