On 2020-01-20 12:48 a.m., Andrew Barnert wrote:

On Jan 19, 2020, at 15:10, Soni L. wrote:

...

We have importlib. We have importlib.resources. We can import modules. We cannot (yet) import resources using the same-ish module import machinery.

First, do you know about setuptools resources? It’s not exactly what you’re looking for, but it has advantages like working automatically for packages installed as zip files, and can be easily extended to do things like i18n- or platform-driven different versions of files, and it works with existing Python.

Meanwhile, if that’s not appropriate or not good enough somehow, couldn’t you do what you want without any new syntax? For example, if you write an import hook that registers a meta for .txt files that just returns a string instead of a module object, wouldn’t `from foo.bar import spam` be able to find foo/bar/spam.txt and bind its contents to spam (and also work with zip files, combined namespace packages,, relative imports, `import foo.bar.spam` and then accessing it as a member of foo.bar, etc.)? And you could likewise register something like `.dat` for binary files, or a whole range of extensions for PNG and OGG and so on, or even a meta that uses file magic to decide whether to import as text or bytes. (Or the hook could even be hookable so your app just says .png or MIME type image/png imports as a PIL.Image and then just imports PNG resource files.)

Of course that’s obviously “messing with importlib”, but it’s something you’d only have to do once, not over and over for each project. You could put it on PyPI and get real life experience showing how it makes code nicer than using setuptools resources and then have a much more compelling case for adding it to Python. And of course it would work as a backport for earlier versions of Python.

What's wrong with the existing import machinery and why does that mean I need setuptools?

...

It would be nice if we could.

I'm thinking of something like:

from foo.bar import resources "foo.txt" as foo, "bar.txt" as bar # string imports from foo.bar import bresources "foo.txt" as bfoo, "bar.txt" as bbar # bytes imports

I’m pretty sure there’s a lot of code out there that uses the name “resources”, so this would be a backward compatibility hit that would probably require the usual 4-1/2 year deprecation and/or future transition.

But do you even need a keyword there? In the current syntax, a quoted string after import is an error, so why not just change that to always mean a (resource) filename without needing a keyword marker? I suppose that doesn’t handle the bytes case, but there might be a better solution there.

Contextual keywords. If not followed by a string, just "fallback" to the old stuff. (The lookahead here is not a problem and you can trivially workaround it in the parser by making the contextual keyword path include (part of) the existing path as a continuation.)

...

or maybe:

foo = f"{from foo.bar import 'foo.txt'}" # string imports bbar = fb"{from foo.bar import 'bar.txt'}" # bytes imports

This version would require turning import from a statement into an expression, which seems like a pretty big change.

It would not. Equivalently, ''!r isn't an expression and yet you can do f"{''!r}", because it's special syntax of fstrings.

And it would be a very weird expression that has side effects but no value when you import a target, but a value and no side effects when you import a string, and… I’m not even sure what when you import a list of strings, much less a mixed list of strings and targets.

You don't even know what it would be (see above) so how am I supposed to make sense of the rest of this argument?

And why does it even need to be a statement? If you just want to get a value; what’s wrong with the function call syntax (a la setuptools)? The only reason import has to be special syntax in the first place is that it does stuff like adding bindings into the outer namespace, and your proposal (in this second version) doesn’t need that.

Don't forget caching. And don't forget to (re-)read up on how it actually works before making claims about it. And the point is that you shouldn't have to import stuff to be able to import stuff, even if the latter stuff is resources. It should be built into the language just like importing modules is built into the language.

On 2020-01-20 1:52 a.m., Andrew Barnert wrote:

On Jan 19, 2020, at 20:13, Soni L. wrote:

...

...

On 2020-01-20 12:48 a.m., Andrew Barnert wrote: On Jan 19, 2020, at 15:10, Soni L. wrote:

...

...

We have importlib. We have importlib.resources. We can import modules. We cannot (yet) import resources using the same-ish module import machinery.

First, do you know about setuptools resources? It’s not exactly what you’re looking for, but it has advantages like working automatically for packages installed as zip files, and can be easily extended to do things like i18n- or platform-driven different versions of files, and it works with existing Python.

Meanwhile, if that’s not appropriate or not good enough somehow, couldn’t you do what you want without any new syntax? For example, if you write an import hook that registers a meta for .txt files that just returns a string instead of a module object, wouldn’t `from foo.bar import spam` be able to find foo/bar/spam.txt and bind its contents to spam (and also work with zip files, combined namespace packages,, relative imports, `import foo.bar.spam` and then accessing it as a member of foo.bar, etc.)? And you could likewise register something like `.dat` for binary files, or a whole range of extensions for PNG and OGG and so on, or even a meta that uses file magic to decide whether to import as text or bytes. (Or the hook could even be hookable so your app just says .png or MIME type image/png imports as a PIL.Image and then just imports PNG resource files.)

Of course that’s obviously “messing with importlib”, but it’s something you’d only have to do once, not over and over for each project. You could put it on PyPI and get real life experience showing how it makes code nicer than using setuptools resources and then have a much more compelling case for adding it to Python. And of course it would work as a backport for earlier versions of Python.

What's wrong with the existing import machinery and why does that mean I need setuptools?

Well, to start with, the existing import machinery doesn’t do what you want, and setuptools does.

If you can put together a concrete proposal that convinces everyone and then wait half a decade for it to be reliably available everywhere, then maybe the existing-in-the-future import machinery will do what you want, but setuptools already does today. Of course you don’t _have_ to use it, but I don’t see why you wouldn’t _want_ to use it.

And meanwhile, setuptools resources also already have features that you haven’t proposed and that would be hard to fit into your proposal.

What the eff is this then? https://docs.python.org/3/library/importlib.html#module-importlib.resources Because I'm pretty sure this is literally part of the existing import machinery. Because importlib *is* the existing import machinery. But what do I know, I'm only a person who reads what it says right there at the top of the module, "The purpose of the importlib package is two-fold. One is to provide the implementation of the import statement (and thus, by extension, the __import__() function) in Python source code. This provides an implementation of import which is portable to any Python interpreter. This also provides an implementation which is easier to comprehend than one implemented in a programming language other than Python. [...]".

...

...

...

It would be nice if we could.

...

I'm thinking of something like: from foo.bar import resources "foo.txt" as foo, "bar.txt" as bar # string imports from foo.bar import bresources "foo.txt" as bfoo, "bar.txt" as bbar # bytes imports

I’m pretty sure there’s a lot of code out there that uses the name “resources”, so this would be a backward compatibility hit that would probably require the usual 4-1/2 year deprecation and/or future transition.

But do you even need a keyword there? In the current syntax, a quoted string after import is an error, so why not just change that to always mean a (resource) filename without needing a keyword marker? I suppose that doesn’t handle the bytes case, but there might be a better solution there.

Contextual keywords.

Which would have all the same problems that all contextual keywords have. People might accept temporarily-contextual keywords that become full keywords after a 3-version transition, for a compelling enough use case like async and await, but not as a permanent part of the grammar for no good reason for a feature that doesn’t add anything you can’t already do without syntax.

Forget this syntax then.

...

...

...

or maybe:

...

foo = f"{from foo.bar import 'foo.txt'}" # string imports bbar = fb"{from foo.bar import 'bar.txt'}" # bytes imports

This version would require turning import from a statement into an expression, which seems like a pretty big change.

It would not. Equivalently, ''!r isn't an expression and yet you can do f"{''!r}", because it's special syntax of fstrings.

That’s not even remotely equivalent. A format replacement_field has an expression and optional conversion and type specifiers. The fact that conversion and type specifiers aren’t expressions doesn’t mean the expression isn’t an expression. That’s like arguing that digits don’t have to have numeric values because you can put a - at the start of a number and - doesn’t have a numeric value.

Plus, you’re clearly trying to use the value of the import, which inherently means it’s an expression. Statements don’t have values. Neither do conversion specifiers, or random fragments of syntax. Only expressions have values.

It wouldn't be an expression in itself, it would be an expression *in a fstring*. fstrings *do not follow normal python syntax* so we can literally add a rule that says "if you see '{from $NAME import $STR}' in an fstring, do this" to the syntax. it doesn't need to become an expression, it just needs to turn the '{...}' part into the desired value. oh and, for that reason, no modifiers or whatnot would be applicable. you do not have to turn it into an expression or define it as an expression. besides, you get different behaviour for unicode fstrings and bytes fstrings which should further help to drive the point home that it's not an expression within an fstring, but a special construct of fstrings themselves. for all I care we could even... add istrings or ibstrings? foo = i"foo.bar:foo.txt" bbar = ib"foo.bar:bar.txt" maybe? altho I much prefer the fstring variants. and if we go with the fstring variants we can also restrict that construct to only fstrings where the construct spans the whole fstring, e.g. f"hey{from foo import 'bar.txt'}" would be an error because of the "hey" at the start.

...

...

And it would be a very weird expression that has side effects but no value when you import a target, but a value and no side effects when you import a string, and… I’m not even sure what when you import a list of strings, much less a mixed list of strings and targets.

You don't even know what it would be (see above) so how am I supposed to make sense of the rest of this argument?

The reason I don’t know what it would be is that there’s nothing sensible for it to be. What do you think is a sensible value of `from foo import bar` is, or of `from foo import bar, “baz.txt”`?

...

...

And why does it even need to be a statement? If you just want to get a value; what’s wrong with the function call syntax (a la setuptools)? The only reason import has to be special syntax in the first place is that it does stuff like adding bindings into the outer namespace, and your proposal (in this second version) doesn’t need that.

Don't forget caching.

What about caching? Do you think functions can’t do caching?

...

And don't forget to (re-)read up on how it actually works before making claims about it.

You haven’t described how it actually works, so what do you want me to re-read?

This "it" refers to the import machinery/"import" itself. e.g. https://docs.python.org/3/library/importlib.html#module-importlib.resources

...

And the point is that you shouldn't have to import stuff to be able to import stuff, If even if the latter stuff is resources. It should be built into the language just like importing modules is built into the language.

Why?

You could make this same claim about anything: you shouldn’t have to import stuff to be able to deserialize stuff with JSON or to do elementwise operations on arrays or whatever. You need some argument for why reading text files is so much more important than all the other things you use a module for that it needs to be builtin.

The reason importing modules is special is the obvious bootstrapping problem: if you had to import a module to import modules, there would be no way to import that first module. That obviously isn’t true for loading text files.

And even then, import could just be a builtin function (the way it is in many other languages); the reason it needs special syntax is that it needs to bind things in the outer scope.

if import was a function then we wouldn't have "from foo import *" and we wouldn't be having this argument. it's not a function tho ;)

On Jan 19, 2020, at 22:22, Soni L. wrote:

...

...

...

...

or maybe:

...

foo = f"{from foo.bar import 'foo.txt'}" # string imports bbar = fb"{from foo.bar import 'bar.txt'}" # bytes imports This version would require turning import from a statement into an expression, which seems like a pretty big change. It would not. Equivalently, ''!r isn't an expression and yet you can do f"{''!r}", because it's special syntax of fstrings.

That’s not even remotely equivalent. A format replacement_field has an expression and optional conversion and type specifiers. The fact that conversion and type specifiers aren’t expressions doesn’t mean the expression isn’t an expression. That’s like arguing that digits don’t have to have numeric values because you can put a - at the start of a number and - doesn’t have a numeric value.

Plus, you’re clearly trying to use the value of the import, which inherently means it’s an expression. Statements don’t have values. Neither do conversion specifiers, or random fragments of syntax. Only expressions have values.

It wouldn't be an expression in itself, it would be an expression *in a fstring*. fstrings *do not follow normal python syntax*

Yes they do. Look at the grammar in the library docs: f_string ::= (literal_char | "{{" | "}}" | replacement_field)* replacement_field ::= "{" f_expression ["!" conversion] [":" format_spec] "}" f_expression ::= (conditional_expression | "*" or_expr) ("," conditional_expression | "," "*" or_expr)* [","] | yield_expression conversion ::= "s" | "r" | "a" format_spec ::= (literal_char | NULL | replacement_field)* literal_char ::=

so we can literally add a rule that says "if you see '{from $NAME import $STR}' in an fstring, do this" to the syntax. it doesn't need to become an expression, it just needs to turn the '{...}' part into the desired value. oh and, for that reason, no modifiers or whatnot would be applicable.

So instead of adding a new expression to Python, you want to change f-strings from being a subset of Python to being a subset of Python plus a new import expression that isn’t in Python but looks confusingly similar to something that is.

you do not have to turn it into an expression or define it as an expression. besides, you get different behaviour for unicode fstrings and bytes fstrings which should further help to drive the point home that it's not an expression within an fstring,

There are no bytes f-strings. I think there are rejected PEPs for both the obvious ways of doing things (add a __bfformat__ protocol, or don’t allow format specs), but they’re both rejected. But even if there were binary f-strings, it wouldn’t show any such thing anyway. The whole reason bytes format makes sense is that string format works by evaluating an expression to get an ordinary value and then calling a method on the result. A bytes format would do the same thing, but call a different method (presumably __bformat__ falling back to __bytes__ instead of __format__ falling back to __str__, although you might also want to add the C buffer protocol as a fallback) on that value.

...

...

And the point is that you shouldn't have to import stuff to be able to import stuff, If even if the latter stuff is resources. It should be built into the language just like importing modules is built into the language.

Why?

You could make this same claim about anything: you shouldn’t have to import stuff to be able to deserialize stuff with JSON or to do elementwise operations on arrays or whatever. You need some argument for why reading text files is so much more important than all the other things you use a module for that it needs to be builtin.

The reason importing modules is special is the obvious bootstrapping problem: if you had to import a module to import modules, there would be no way to import that first module. That obviously isn’t true for loading text files.

And even then, import could just be a builtin function (the way it is in many other languages); the reason it needs special syntax is that it needs to bind things in the outer scope.

if import was a function then we wouldn't have "from foo import *" and we wouldn't be having this argument. it's not a function tho ;)

I’m sorry, but this doesn’t even make sense as a reply to what I said. I explained why import can’t be a function. But your resource import-expression (or import-not-an-expression-but-f-strings-treat-them-as-if-they-we’re) can be a function, because it doesn’t need to bind any names or do anything else with the containing scope. So you have to explain why it needs custom syntax instead of just being a function. You also need to explain why it even needs to be builtin rather than imported, even though being imported is fine for tons of other useful stuff that people do even more often. If you can’t explain either of those things, and if imporlib.reaources already does everything you want, the answer is to just import it and use it.