We have importlib. We have importlib.resources. We can import modules. We cannot (yet) import resources using the same-ish module import machinery. It would be nice if we could. I'm thinking of something like: from foo.bar import resources "foo.txt" as foo, "bar.txt" as bar # string imports from foo.bar import bresources "foo.txt" as bfoo, "bar.txt" as bbar # bytes imports or maybe: foo = f"{from foo.bar import 'foo.txt'}" # string imports bbar = fb"{from foo.bar import 'bar.txt'}" # bytes imports altho I have a bit of a preference for the latter. I feel like these would be nice, and useful for packaging default configs, templates, and whatnot: def unpack_defaults(): open("templates/index.html", "w").write(f"{from ganarchy.templates import 'index.html'}") open("templates/project.html", "w").write(f"{from ganarchy.templates import 'project.html'}") open("templates/config.toml", "w").write(f"{from ganarchy.templates import 'config.toml'}") You'd just do this, effortlessly, no need to mess with importlib, it would just work and it'd be easy to teach ppl and get them to use it.
On Jan 19, 2020, at 15:10, Soni L. <fakedme+py@gmail.com> wrote:
We have importlib. We have importlib.resources. We can import modules. We cannot (yet) import resources using the same-ish module import machinery.
First, do you know about setuptools resources? It’s not exactly what you’re looking for, but it has advantages like working automatically for packages installed as zip files, and can be easily extended to do things like i18n- or platform-driven different versions of files, and it works with existing Python. Meanwhile, if that’s not appropriate or not good enough somehow, couldn’t you do what you want without any new syntax? For example, if you write an import hook that registers a meta for .txt files that just returns a string instead of a module object, wouldn’t `from foo.bar import spam` be able to find foo/bar/spam.txt and bind its contents to spam (and also work with zip files, combined namespace packages,, relative imports, `import foo.bar.spam` and then accessing it as a member of foo.bar, etc.)? And you could likewise register something like `.dat` for binary files, or a whole range of extensions for PNG and OGG and so on, or even a meta that uses file magic to decide whether to import as text or bytes. (Or the hook could even be hookable so your app just says .png or MIME type image/png imports as a PIL.Image and then just imports PNG resource files.) Of course that’s obviously “messing with importlib”, but it’s something you’d only have to do once, not over and over for each project. You could put it on PyPI and get real life experience showing how it makes code nicer than using setuptools resources and then have a much more compelling case for adding it to Python. And of course it would work as a backport for earlier versions of Python.
I’m pretty sure there’s a lot of code out there that uses the name “resources”, so this would be a backward compatibility hit that would probably require the usual 4-1/2 year deprecation and/or future transition. But do you even need a keyword there? In the current syntax, a quoted string after import is an error, so why not just change that to always mean a (resource) filename without needing a keyword marker? I suppose that doesn’t handle the bytes case, but there might be a better solution there.
This version would require turning import from a statement into an expression, which seems like a pretty big change. And it would be a very weird expression that has side effects but no value when you import a target, but a value and no side effects when you import a string, and… I’m not even sure what when you import a list of strings, much less a mixed list of strings and targets. And why does it even need to be a statement? If you just want to get a value; what’s wrong with the function call syntax (a la setuptools)? The only reason import has to be special syntax in the first place is that it does stuff like adding bindings into the outer namespace, and your proposal (in this second version) doesn’t need that.
On 2020-01-20 12:48 a.m., Andrew Barnert wrote:
What's wrong with the existing import machinery and why does that mean I need setuptools?
Contextual keywords. If not followed by a string, just "fallback" to the old stuff. (The lookahead here is not a problem and you can trivially workaround it in the parser by making the contextual keyword path include (part of) the existing path as a continuation.)
It would not. Equivalently, ''!r isn't an expression and yet you can do f"{''!r}", because it's special syntax of fstrings.
And it would be a very weird expression that has side effects but no value when you import a target, but a value and no side effects when you import a string, and… I’m not even sure what when you import a list of strings, much less a mixed list of strings and targets.
You don't even know what it would be (see above) so how am I supposed to make sense of the rest of this argument?
And why does it even need to be a statement? If you just want to get a value; what’s wrong with the function call syntax (a la setuptools)? The only reason import has to be special syntax in the first place is that it does stuff like adding bindings into the outer namespace, and your proposal (in this second version) doesn’t need that.
Don't forget caching. And don't forget to (re-)read up on how it actually works before making claims about it. And the point is that you shouldn't have to import stuff to be able to import stuff, even if the latter stuff is resources. It should be built into the language just like importing modules is built into the language.
On Jan 19, 2020, at 20:13, Soni L. <fakedme+py@gmail.com> wrote:
Well, to start with, the existing import machinery doesn’t do what you want, and setuptools does. If you can put together a concrete proposal that convinces everyone and then wait half a decade for it to be reliably available everywhere, then maybe the existing-in-the-future import machinery will do what you want, but setuptools already does today. Of course you don’t _have_ to use it, but I don’t see why you wouldn’t _want_ to use it. And meanwhile, setuptools resources also already have features that you haven’t proposed and that would be hard to fit into your proposal.
Which would have all the same problems that all contextual keywords have. People might accept temporarily-contextual keywords that become full keywords after a 3-version transition, for a compelling enough use case like async and await, but not as a permanent part of the grammar for no good reason for a feature that doesn’t add anything you can’t already do without syntax.
That’s not even remotely equivalent. A format replacement_field has an expression and optional conversion and type specifiers. The fact that conversion and type specifiers aren’t expressions doesn’t mean the expression isn’t an expression. That’s like arguing that digits don’t have to have numeric values because you can put a - at the start of a number and - doesn’t have a numeric value. Plus, you’re clearly trying to use the value of the import, which inherently means it’s an expression. Statements don’t have values. Neither do conversion specifiers, or random fragments of syntax. Only expressions have values.
And it would be a very weird expression that has side effects but no value when you import a target, but a value and no side effects when you import a string, and… I’m not even sure what when you import a list of strings, much less a mixed list of strings and targets.
You don't even know what it would be (see above) so how am I supposed to make sense of the rest of this argument?
The reason I don’t know what it would be is that there’s nothing sensible for it to be. What do you think is a sensible value of `from foo import bar` is, or of `from foo import bar, “baz.txt”`?
And why does it even need to be a statement? If you just want to get a value; what’s wrong with the function call syntax (a la setuptools)? The only reason import has to be special syntax in the first place is that it does stuff like adding bindings into the outer namespace, and your proposal (in this second version) doesn’t need that.
Don't forget caching.
What about caching? Do you think functions can’t do caching?
And don't forget to (re-)read up on how it actually works before making claims about it.
You haven’t described how it actually works, so what do you want me to re-read?
And the point is that you shouldn't have to import stuff to be able to import stuff, If even if the latter stuff is resources. It should be built into the language just like importing modules is built into the language.
Why? You could make this same claim about anything: you shouldn’t have to import stuff to be able to deserialize stuff with JSON or to do elementwise operations on arrays or whatever. You need some argument for why reading text files is so much more important than all the other things you use a module for that it needs to be builtin. The reason importing modules is special is the obvious bootstrapping problem: if you had to import a module to import modules, there would be no way to import that first module. That obviously isn’t true for loading text files. And even then, import could just be a builtin function (the way it is in many other languages); the reason it needs special syntax is that it needs to bind things in the outer scope.
On 2020-01-20 1:52 a.m., Andrew Barnert wrote:
What the eff is this then? https://docs.python.org/3/library/importlib.html#module-importlib.resources Because I'm pretty sure this is literally part of the existing import machinery. Because importlib *is* the existing import machinery. But what do I know, I'm only a person who reads what it says right there at the top of the module, "The purpose of the importlib package is two-fold. One is to provide the implementation of the import statement (and thus, by extension, the __import__() function) in Python source code. This provides an implementation of import which is portable to any Python interpreter. This also provides an implementation which is easier to comprehend than one implemented in a programming language other than Python. [...]".
Forget this syntax then.
It wouldn't be an expression in itself, it would be an expression *in a fstring*. fstrings *do not follow normal python syntax* so we can literally add a rule that says "if you see '{from $NAME import $STR}' in an fstring, do this" to the syntax. it doesn't need to become an expression, it just needs to turn the '{...}' part into the desired value. oh and, for that reason, no modifiers or whatnot would be applicable. you do not have to turn it into an expression or define it as an expression. besides, you get different behaviour for unicode fstrings and bytes fstrings which should further help to drive the point home that it's not an expression within an fstring, but a special construct of fstrings themselves. for all I care we could even... add istrings or ibstrings? foo = i"foo.bar:foo.txt" bbar = ib"foo.bar:bar.txt" maybe? altho I much prefer the fstring variants. and if we go with the fstring variants we can also restrict that construct to only fstrings where the construct spans the whole fstring, e.g. f"hey{from foo import 'bar.txt'}" would be an error because of the "hey" at the start.
This "it" refers to the import machinery/"import" itself. e.g. https://docs.python.org/3/library/importlib.html#module-importlib.resources
if import was a function then we wouldn't have "from foo import *" and we wouldn't be having this argument. it's not a function tho ;)
On Mon, Jan 20, 2020 at 5:23 PM Soni L. <fakedme+py@gmail.com> wrote:
No need to get caustic. Remember, not everyone is aware of everything that the standard library has. To my knowledge, this is the first time that importlib.resources has been mentioned in this thread, and you proposed something very similar as though it were a new feature, without any reference to this module.
I don't understand the point of having this in an f-string. Are you going to do constructs like: response = bf"""HTTP/1.0 404 NOT FOUND {headers} {from foo.bar import 'foo.txt'} """ where you combine these magic imports with other content? If not, why not just use a simple function call? Lately, I've been seeing a number of python-ideas posts that have the feeling of "this is a new toy, let's use it in even more places". Just because := and f"..." exist, they have to be used for everything?
if import was a function then we wouldn't have "from foo import *" and we wouldn't be having this argument. it's not a function tho ;)
__import__("sys") <module 'sys' (built-in)>
:) ChrisA
On Jan 19, 2020, at 22:22, Soni L. <fakedme+py@gmail.com> wrote:
Yes they do. Look at the grammar in the library docs: f_string ::= (literal_char | "{{" | "}}" | replacement_field)* replacement_field ::= "{" f_expression ["!" conversion] [":" format_spec] "}" f_expression ::= (conditional_expression | "*" or_expr) ("," conditional_expression | "," "*" or_expr)* [","] | yield_expression conversion ::= "s" | "r" | "a" format_spec ::= (literal_char | NULL | replacement_field)* literal_char ::= <any code point except The f_expression is a subset of the same expression used by the main syntax. All those alternate productions it links to are the ones in the main syntax. The syntax for the expression part of an f-string is an expression. And look at the semantics. What it formats is the result of evaluating that expression (as normal Python code in the relevant scope).
so we can literally add a rule that says "if you see '{from $NAME import $STR}' in an fstring, do this" to the syntax. it doesn't need to become an expression, it just needs to turn the '{...}' part into the desired value. oh and, for that reason, no modifiers or whatnot would be applicable.
So instead of adding a new expression to Python, you want to change f-strings from being a subset of Python to being a subset of Python plus a new import expression that isn’t in Python but looks confusingly similar to something that is.
you do not have to turn it into an expression or define it as an expression. besides, you get different behaviour for unicode fstrings and bytes fstrings which should further help to drive the point home that it's not an expression within an fstring,
There are no bytes f-strings. I think there are rejected PEPs for both the obvious ways of doing things (add a __bfformat__ protocol, or don’t allow format specs), but they’re both rejected. But even if there were binary f-strings, it wouldn’t show any such thing anyway. The whole reason bytes format makes sense is that string format works by evaluating an expression to get an ordinary value and then calling a method on the result. A bytes format would do the same thing, but call a different method (presumably __bformat__ falling back to __bytes__ instead of __format__ falling back to __str__, although you might also want to add the C buffer protocol as a fallback) on that value.
I’m sorry, but this doesn’t even make sense as a reply to what I said. I explained why import can’t be a function. But your resource import-expression (or import-not-an-expression-but-f-strings-treat-them-as-if-they-we’re) can be a function, because it doesn’t need to bind any names or do anything else with the containing scope. So you have to explain why it needs custom syntax instead of just being a function. You also need to explain why it even needs to be builtin rather than imported, even though being imported is fine for tons of other useful stuff that people do even more often. If you can’t explain either of those things, and if imporlib.reaources already does everything you want, the answer is to just import it and use it.
On Jan 19, 2020, at 15:10, Soni L. <fakedme+py@gmail.com> wrote:
We have importlib. We have importlib.resources. We can import modules. We cannot (yet) import resources using the same-ish module import machinery.
First, do you know about setuptools resources? It’s not exactly what you’re looking for, but it has advantages like working automatically for packages installed as zip files, and can be easily extended to do things like i18n- or platform-driven different versions of files, and it works with existing Python. Meanwhile, if that’s not appropriate or not good enough somehow, couldn’t you do what you want without any new syntax? For example, if you write an import hook that registers a meta for .txt files that just returns a string instead of a module object, wouldn’t `from foo.bar import spam` be able to find foo/bar/spam.txt and bind its contents to spam (and also work with zip files, combined namespace packages,, relative imports, `import foo.bar.spam` and then accessing it as a member of foo.bar, etc.)? And you could likewise register something like `.dat` for binary files, or a whole range of extensions for PNG and OGG and so on, or even a meta that uses file magic to decide whether to import as text or bytes. (Or the hook could even be hookable so your app just says .png or MIME type image/png imports as a PIL.Image and then just imports PNG resource files.) Of course that’s obviously “messing with importlib”, but it’s something you’d only have to do once, not over and over for each project. You could put it on PyPI and get real life experience showing how it makes code nicer than using setuptools resources and then have a much more compelling case for adding it to Python. And of course it would work as a backport for earlier versions of Python.
I’m pretty sure there’s a lot of code out there that uses the name “resources”, so this would be a backward compatibility hit that would probably require the usual 4-1/2 year deprecation and/or future transition. But do you even need a keyword there? In the current syntax, a quoted string after import is an error, so why not just change that to always mean a (resource) filename without needing a keyword marker? I suppose that doesn’t handle the bytes case, but there might be a better solution there.
This version would require turning import from a statement into an expression, which seems like a pretty big change. And it would be a very weird expression that has side effects but no value when you import a target, but a value and no side effects when you import a string, and… I’m not even sure what when you import a list of strings, much less a mixed list of strings and targets. And why does it even need to be a statement? If you just want to get a value; what’s wrong with the function call syntax (a la setuptools)? The only reason import has to be special syntax in the first place is that it does stuff like adding bindings into the outer namespace, and your proposal (in this second version) doesn’t need that.
On 2020-01-20 12:48 a.m., Andrew Barnert wrote:
What's wrong with the existing import machinery and why does that mean I need setuptools?
Contextual keywords. If not followed by a string, just "fallback" to the old stuff. (The lookahead here is not a problem and you can trivially workaround it in the parser by making the contextual keyword path include (part of) the existing path as a continuation.)
It would not. Equivalently, ''!r isn't an expression and yet you can do f"{''!r}", because it's special syntax of fstrings.
And it would be a very weird expression that has side effects but no value when you import a target, but a value and no side effects when you import a string, and… I’m not even sure what when you import a list of strings, much less a mixed list of strings and targets.
You don't even know what it would be (see above) so how am I supposed to make sense of the rest of this argument?
And why does it even need to be a statement? If you just want to get a value; what’s wrong with the function call syntax (a la setuptools)? The only reason import has to be special syntax in the first place is that it does stuff like adding bindings into the outer namespace, and your proposal (in this second version) doesn’t need that.
Don't forget caching. And don't forget to (re-)read up on how it actually works before making claims about it. And the point is that you shouldn't have to import stuff to be able to import stuff, even if the latter stuff is resources. It should be built into the language just like importing modules is built into the language.
On Jan 19, 2020, at 20:13, Soni L. <fakedme+py@gmail.com> wrote:
Well, to start with, the existing import machinery doesn’t do what you want, and setuptools does. If you can put together a concrete proposal that convinces everyone and then wait half a decade for it to be reliably available everywhere, then maybe the existing-in-the-future import machinery will do what you want, but setuptools already does today. Of course you don’t _have_ to use it, but I don’t see why you wouldn’t _want_ to use it. And meanwhile, setuptools resources also already have features that you haven’t proposed and that would be hard to fit into your proposal.
Which would have all the same problems that all contextual keywords have. People might accept temporarily-contextual keywords that become full keywords after a 3-version transition, for a compelling enough use case like async and await, but not as a permanent part of the grammar for no good reason for a feature that doesn’t add anything you can’t already do without syntax.
That’s not even remotely equivalent. A format replacement_field has an expression and optional conversion and type specifiers. The fact that conversion and type specifiers aren’t expressions doesn’t mean the expression isn’t an expression. That’s like arguing that digits don’t have to have numeric values because you can put a - at the start of a number and - doesn’t have a numeric value. Plus, you’re clearly trying to use the value of the import, which inherently means it’s an expression. Statements don’t have values. Neither do conversion specifiers, or random fragments of syntax. Only expressions have values.
And it would be a very weird expression that has side effects but no value when you import a target, but a value and no side effects when you import a string, and… I’m not even sure what when you import a list of strings, much less a mixed list of strings and targets.
You don't even know what it would be (see above) so how am I supposed to make sense of the rest of this argument?
The reason I don’t know what it would be is that there’s nothing sensible for it to be. What do you think is a sensible value of `from foo import bar` is, or of `from foo import bar, “baz.txt”`?
And why does it even need to be a statement? If you just want to get a value; what’s wrong with the function call syntax (a la setuptools)? The only reason import has to be special syntax in the first place is that it does stuff like adding bindings into the outer namespace, and your proposal (in this second version) doesn’t need that.
Don't forget caching.
What about caching? Do you think functions can’t do caching?
And don't forget to (re-)read up on how it actually works before making claims about it.
You haven’t described how it actually works, so what do you want me to re-read?
And the point is that you shouldn't have to import stuff to be able to import stuff, If even if the latter stuff is resources. It should be built into the language just like importing modules is built into the language.
Why? You could make this same claim about anything: you shouldn’t have to import stuff to be able to deserialize stuff with JSON or to do elementwise operations on arrays or whatever. You need some argument for why reading text files is so much more important than all the other things you use a module for that it needs to be builtin. The reason importing modules is special is the obvious bootstrapping problem: if you had to import a module to import modules, there would be no way to import that first module. That obviously isn’t true for loading text files. And even then, import could just be a builtin function (the way it is in many other languages); the reason it needs special syntax is that it needs to bind things in the outer scope.
On 2020-01-20 1:52 a.m., Andrew Barnert wrote:
What the eff is this then? https://docs.python.org/3/library/importlib.html#module-importlib.resources Because I'm pretty sure this is literally part of the existing import machinery. Because importlib *is* the existing import machinery. But what do I know, I'm only a person who reads what it says right there at the top of the module, "The purpose of the importlib package is two-fold. One is to provide the implementation of the import statement (and thus, by extension, the __import__() function) in Python source code. This provides an implementation of import which is portable to any Python interpreter. This also provides an implementation which is easier to comprehend than one implemented in a programming language other than Python. [...]".
Forget this syntax then.
It wouldn't be an expression in itself, it would be an expression *in a fstring*. fstrings *do not follow normal python syntax* so we can literally add a rule that says "if you see '{from $NAME import $STR}' in an fstring, do this" to the syntax. it doesn't need to become an expression, it just needs to turn the '{...}' part into the desired value. oh and, for that reason, no modifiers or whatnot would be applicable. you do not have to turn it into an expression or define it as an expression. besides, you get different behaviour for unicode fstrings and bytes fstrings which should further help to drive the point home that it's not an expression within an fstring, but a special construct of fstrings themselves. for all I care we could even... add istrings or ibstrings? foo = i"foo.bar:foo.txt" bbar = ib"foo.bar:bar.txt" maybe? altho I much prefer the fstring variants. and if we go with the fstring variants we can also restrict that construct to only fstrings where the construct spans the whole fstring, e.g. f"hey{from foo import 'bar.txt'}" would be an error because of the "hey" at the start.
This "it" refers to the import machinery/"import" itself. e.g. https://docs.python.org/3/library/importlib.html#module-importlib.resources
if import was a function then we wouldn't have "from foo import *" and we wouldn't be having this argument. it's not a function tho ;)
On Mon, Jan 20, 2020 at 5:23 PM Soni L. <fakedme+py@gmail.com> wrote:
No need to get caustic. Remember, not everyone is aware of everything that the standard library has. To my knowledge, this is the first time that importlib.resources has been mentioned in this thread, and you proposed something very similar as though it were a new feature, without any reference to this module.
I don't understand the point of having this in an f-string. Are you going to do constructs like: response = bf"""HTTP/1.0 404 NOT FOUND {headers} {from foo.bar import 'foo.txt'} """ where you combine these magic imports with other content? If not, why not just use a simple function call? Lately, I've been seeing a number of python-ideas posts that have the feeling of "this is a new toy, let's use it in even more places". Just because := and f"..." exist, they have to be used for everything?
if import was a function then we wouldn't have "from foo import *" and we wouldn't be having this argument. it's not a function tho ;)
__import__("sys") <module 'sys' (built-in)>
:) ChrisA
On Jan 19, 2020, at 22:22, Soni L. <fakedme+py@gmail.com> wrote:
Yes they do. Look at the grammar in the library docs: f_string ::= (literal_char | "{{" | "}}" | replacement_field)* replacement_field ::= "{" f_expression ["!" conversion] [":" format_spec] "}" f_expression ::= (conditional_expression | "*" or_expr) ("," conditional_expression | "," "*" or_expr)* [","] | yield_expression conversion ::= "s" | "r" | "a" format_spec ::= (literal_char | NULL | replacement_field)* literal_char ::= <any code point except The f_expression is a subset of the same expression used by the main syntax. All those alternate productions it links to are the ones in the main syntax. The syntax for the expression part of an f-string is an expression. And look at the semantics. What it formats is the result of evaluating that expression (as normal Python code in the relevant scope).
so we can literally add a rule that says "if you see '{from $NAME import $STR}' in an fstring, do this" to the syntax. it doesn't need to become an expression, it just needs to turn the '{...}' part into the desired value. oh and, for that reason, no modifiers or whatnot would be applicable.
So instead of adding a new expression to Python, you want to change f-strings from being a subset of Python to being a subset of Python plus a new import expression that isn’t in Python but looks confusingly similar to something that is.
you do not have to turn it into an expression or define it as an expression. besides, you get different behaviour for unicode fstrings and bytes fstrings which should further help to drive the point home that it's not an expression within an fstring,
There are no bytes f-strings. I think there are rejected PEPs for both the obvious ways of doing things (add a __bfformat__ protocol, or don’t allow format specs), but they’re both rejected. But even if there were binary f-strings, it wouldn’t show any such thing anyway. The whole reason bytes format makes sense is that string format works by evaluating an expression to get an ordinary value and then calling a method on the result. A bytes format would do the same thing, but call a different method (presumably __bformat__ falling back to __bytes__ instead of __format__ falling back to __str__, although you might also want to add the C buffer protocol as a fallback) on that value.
I’m sorry, but this doesn’t even make sense as a reply to what I said. I explained why import can’t be a function. But your resource import-expression (or import-not-an-expression-but-f-strings-treat-them-as-if-they-we’re) can be a function, because it doesn’t need to bind any names or do anything else with the containing scope. So you have to explain why it needs custom syntax instead of just being a function. You also need to explain why it even needs to be builtin rather than imported, even though being imported is fine for tons of other useful stuff that people do even more often. If you can’t explain either of those things, and if imporlib.reaources already does everything you want, the answer is to just import it and use it.
participants (3)
-
Andrew Barnert -
Chris Angelico -
Soni L.