On Tue, Oct 19, 2021 at 9:00 AM Cameron Simpson <cs@cskk.id.au> wrote:

The problem with a "download()" method is that it is almost never what you need. There are too many ways to want to do it, and one almost _never_ wants to suck the download itself into memory as you do above with read() because downloads are often large, sometimes very large.

You also don't always want to put it into a file.

OTOH, if you *do* want to put it into a file, it should be possible to take advantage of zero-copy APIs to reduce unnecessary transfers. I'm not sure if there's a way to do that with requests. Ideally, what you want is os.sendfile() but it'd need to be cleanly wrapped by the library itself. ChrisA

On 19/10/2021 00.06, Chris Angelico wrote:

On Tue, Oct 19, 2021 at 9:00 AM Cameron Simpson <cs@cskk.id.au> wrote:

...

The problem with a "download()" method is that it is almost never what you need. There are too many ways to want to do it, and one almost _never_ wants to suck the download itself into memory as you do above with read() because downloads are often large, sometimes very large.

You also don't always want to put it into a file.

OTOH, if you *do* want to put it into a file, it should be possible to take advantage of zero-copy APIs to reduce unnecessary transfers. I'm not sure if there's a way to do that with requests. Ideally, what you want is os.sendfile() but it'd need to be cleanly wrapped by the library itself.

Splicing APIs like sendfile() require a Kernel socket. You cannot do sendfile() with userspace sockets like OpenSSL socket for e.g. HTTPS. Latest Linux Kernel and OpenSSL 3.0.0 have a new feature called kTLS. Kernel TLS uses OpenSSL to establish the TLS connection and then handles payload transfer in Kernel to for zero-copy sendfile(). Christian

I am aware of requests (or httpx, ...), but the idea is to do all that with the standard library. If I have to install stuff, I could just install wget/curl and be done. Feature creep is an issue here, but just like http.server one could be really strict about just covering 90% of the use cases (download stuff and print or save it) and not trying to handle any corner cases. The first code snippet was not supposed to be production-ready. Here's an improved version which only downloads 1MB a time and prints it. The only parameter could be the URL: from urllib.request import urlopen from sys import stdout with urlopen("https://coherentminds.de/") as response: while data := response.read(1024 * 1024): stdout.buffer.write(data) The user of this function could still decide to divert stdout into a file, so both use cases printing and saving would be covered. IMHO, the benefit-cost ratio is quite good: * can be a lifesaver (just like http.server) every once in a while in particular in a container or testing context * low implementation effort * easy to test and to maintain Tom

You are absolutely right, the functionality is there, but the idea is to make it easily available from the command line. Here is a line (with shortened URL) from a Dockerfile which installs poetry as suggested in the docs: RUN python -c "from urllib.request import urlopen; print(urlopen('https://.../install-poetry.py').read().decode())" | python With the proposed functionality, the urllib could provide a download entry point which would make the line look like this: RUN python -m urrlib.download "https://.../install-poetry.py" | python This is less error-prone, would also work for rather large downloads and wouldn't be much effort to implement/test/maintain.

Thanks, finally a +1! \o/ It's funny that "entry point" triggered your reaction, because I think it's not the correct technical term. What I'm proposing is very similar to http.server: https://github.com/python/cpython/blob/3.10/Lib/http/server.py#L1257 Just like "python -m http.server" you could then use "python -m urllib.request.download <URL>".

On Fri, Oct 22, 2021 at 12:37:11PM -0000, Tom Pohl wrote:

A question for the Python experts: What is the correct technical term for a functionality like "http.server", i.e., a module with an actual "main" function?

"A module which can be imported as a library, and also executed as a script or application." -- Steve

Thanks. Not as catchy as I would have hoped, though. ;-) When you respond to a message, could you keep a little of the context

On 10/25/2021 11:21 AM, Tom Pohl wrote: that you're replying to? I'm not sure what this refers to.

One person except me in favor of this idea. Any other feedback? How to proceed?

I think it's a good idea. I think the next step would be to create a PR, including documentation. IIRC, there's already an issue open for this. Eric

Tom Pohl wrote:

A question for the Python experts: What is the correct technical term for a functionality like "http.server", i.e., a module with an actual "main" function?

There's some details about it here https://docs.python.org/3/library/__main__.html#idiomatic-usage

On Tue, Oct 19, 2021 at 9:00 AM Cameron Simpson <cs@cskk.id.au> wrote:

The problem with a "download()" method is that it is almost never what you need. There are too many ways to want to do it, and one almost _never_ wants to suck the download itself into memory as you do above with read() because downloads are often large, sometimes very large.

You also don't always want to put it into a file.

OTOH, if you *do* want to put it into a file, it should be possible to take advantage of zero-copy APIs to reduce unnecessary transfers. I'm not sure if there's a way to do that with requests. Ideally, what you want is os.sendfile() but it'd need to be cleanly wrapped by the library itself. ChrisA

On 19/10/2021 00.06, Chris Angelico wrote:

On Tue, Oct 19, 2021 at 9:00 AM Cameron Simpson <cs@cskk.id.au> wrote:

...

The problem with a "download()" method is that it is almost never what you need. There are too many ways to want to do it, and one almost _never_ wants to suck the download itself into memory as you do above with read() because downloads are often large, sometimes very large.

You also don't always want to put it into a file.

OTOH, if you *do* want to put it into a file, it should be possible to take advantage of zero-copy APIs to reduce unnecessary transfers. I'm not sure if there's a way to do that with requests. Ideally, what you want is os.sendfile() but it'd need to be cleanly wrapped by the library itself.

Splicing APIs like sendfile() require a Kernel socket. You cannot do sendfile() with userspace sockets like OpenSSL socket for e.g. HTTPS. Latest Linux Kernel and OpenSSL 3.0.0 have a new feature called kTLS. Kernel TLS uses OpenSSL to establish the TLS connection and then handles payload transfer in Kernel to for zero-copy sendfile(). Christian

I am aware of requests (or httpx, ...), but the idea is to do all that with the standard library. If I have to install stuff, I could just install wget/curl and be done. Feature creep is an issue here, but just like http.server one could be really strict about just covering 90% of the use cases (download stuff and print or save it) and not trying to handle any corner cases. The first code snippet was not supposed to be production-ready. Here's an improved version which only downloads 1MB a time and prints it. The only parameter could be the URL: from urllib.request import urlopen from sys import stdout with urlopen("https://coherentminds.de/") as response: while data := response.read(1024 * 1024): stdout.buffer.write(data) The user of this function could still decide to divert stdout into a file, so both use cases printing and saving would be covered. IMHO, the benefit-cost ratio is quite good: * can be a lifesaver (just like http.server) every once in a while in particular in a container or testing context * low implementation effort * easy to test and to maintain Tom

You are absolutely right, the functionality is there, but the idea is to make it easily available from the command line. Here is a line (with shortened URL) from a Dockerfile which installs poetry as suggested in the docs: RUN python -c "from urllib.request import urlopen; print(urlopen('https://.../install-poetry.py').read().decode())" | python With the proposed functionality, the urllib could provide a download entry point which would make the line look like this: RUN python -m urrlib.download "https://.../install-poetry.py" | python This is less error-prone, would also work for rather large downloads and wouldn't be much effort to implement/test/maintain.

Thanks, finally a +1! \o/ It's funny that "entry point" triggered your reaction, because I think it's not the correct technical term. What I'm proposing is very similar to http.server: https://github.com/python/cpython/blob/3.10/Lib/http/server.py#L1257 Just like "python -m http.server" you could then use "python -m urllib.request.download <URL>".

On Fri, Oct 22, 2021 at 12:37:11PM -0000, Tom Pohl wrote:

A question for the Python experts: What is the correct technical term for a functionality like "http.server", i.e., a module with an actual "main" function?

"A module which can be imported as a library, and also executed as a script or application." -- Steve

Thanks. Not as catchy as I would have hoped, though. ;-) When you respond to a message, could you keep a little of the context

On 10/25/2021 11:21 AM, Tom Pohl wrote: that you're replying to? I'm not sure what this refers to.

One person except me in favor of this idea. Any other feedback? How to proceed?

I think it's a good idea. I think the next step would be to create a PR, including documentation. IIRC, there's already an issue open for this. Eric

Tom Pohl wrote:

A question for the Python experts: What is the correct technical term for a functionality like "http.server", i.e., a module with an actual "main" function?

There's some details about it here https://docs.python.org/3/library/__main__.html#idiomatic-usage