Add support for private variables, methods and functions in Python
Private methods, functions and variables are very common in programming languages. But Python doesn't support private. It has conventions for naming so considered private but not private. Most of the time private is never required, what Python provides is more than enough. But the need for private come into place when we're dealing with passphrases and servers. For example consider this code, class A: def get(): // code to get the password self.password = password Now consider this,
x = A(); x.get(); x.password
See what just happened? Someone just got the member variable value that the person wasn't supposed to. I suggest to add private support for functions (module __all__ methods to be more clear), methods and variables (module __all__ variables or class variables). (I very bad at reading PEPs so I may miss out something critical that's been explained already (sometimes I miss out a feature in a PEP and think about suggesting that feature when it's already there and then I realize "Oh! It's already here in this PEP"). If that's the case then please correct me.) With Regards, Shreyan Avigyan
Hey Shreyan, From what I understand, Python's design focuses on enabling the developer to do whatever she feels right. Having a class attribute or something marked as "private" is more of a warning to people _using_ that class that they shouldn't tinker with that field. If I'm using a class that deals with a bank account, for example, and the current balance is marked as "private" - or prefixed with __, in Python - I should read that as "well, this value has some elaborated logic behind it. I should not go around messing with it because there may be consequences that this class's author didn't want" and not "I shouldn't be able to see this value!" This is not an information security aspect, it is more of a code usability thing. That said,
Someone just got the member variable value that the person wasn't supposed to
If they can't access a piece of data, they shouldn't be allowed to execute code inside that server to begin with. There's actually a whole class of security flaws that deal with code execution. Once you allow someone to run code on your server, you're done and no keyword in the code will prevent bad things from happening. Best, --Bidu On Wed, May 5, 2021 at 10:41 AM Shreyan Avigyan <pythonshreyan09@gmail.com> wrote:
Private methods, functions and variables are very common in programming languages. But Python doesn't support private. It has conventions for naming so considered private but not private. Most of the time private is never required, what Python provides is more than enough. But the need for private come into place when we're dealing with passphrases and servers. For example consider this code,
class A: def get(): // code to get the password self.password = password
Now consider this,
x = A(); x.get(); x.password
See what just happened? Someone just got the member variable value that the person wasn't supposed to.
I suggest to add private support for functions (module __all__ methods to be more clear), methods and variables (module __all__ variables or class variables).
(I very bad at reading PEPs so I may miss out something critical that's been explained already (sometimes I miss out a feature in a PEP and think about suggesting that feature when it's already there and then I realize "Oh! It's already here in this PEP"). If that's the case then please correct me.)
With Regards, Shreyan Avigyan _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/DD2L56... Code of Conduct: http://python.org/psf/codeofconduct/
On Wed, May 5, 2021 at 11:43 PM Shreyan Avigyan <pythonshreyan09@gmail.com> wrote:
Private methods, functions and variables are very common in programming languages. But Python doesn't support private. It has conventions for naming so considered private but not private. Most of the time private is never required, what Python provides is more than enough. But the need for private come into place when we're dealing with passphrases and servers. For example consider this code,
class A: def get(): // code to get the password self.password = password
Now consider this,
x = A(); x.get(); x.password
See what just happened? Someone just got the member variable value that the person wasn't supposed to.
That's only a problem if you have untrusted code running in the same Python interpreter as privileged code. There's fundamentally no way that you can get the password without the untrusted code also being able to get it. Do you have an actual use-case where you have passphrases like this? How does the "code to get the password" do it, and how do you ensure that the unprivileged code can't just do the same thing? ChrisA
I don't know if it's worth adding private to python modules but what about classes? Private in class is an important feature in OOP that Python lacks (Most OOP languages have private like Java, C++, etc.). I don't know why it was introduced in the first place but it's kind of a convention to have that feature in OOP. Moreover why does private exist in the first place? If there's a reason then shouldn't it apply to Python as well. And yes you're right the unprivileged code can just do the same thing. But I'm not only speaking of security in this context.
Hi Shreyan, You say: On Wed, May 05, 2021 at 02:18:55PM -0000, Shreyan Avigyan wrote:
I don't know if it's worth adding private to python modules but what about classes? Private in class is an important feature in OOP that Python lacks (Most OOP languages have private like Java, C++, etc.). I don't know why it was introduced in the first place but it's kind of a convention to have that feature in OOP.
If you don't know why private was introduced, what makes you think it is a good feature to have? Programmers in languages with "private, protected" etc usually end up spending large amounts of time fighting with the compiler, because some over-zealous other programmers made things private that shouldn't have been. Python finds a nice middle ground between languages which offer no protection and those that encourage too much unnecessary protection. We have a very strong convention not to rely on or use other people's underscore variables, while still allowing people to inspect and even modify those variables if necessary; but if you really, truly want to hide the existence of class attributes you can write the class in C. Explain to us, what *exactly* are you worried about? Suppose you write a class, and have an underscore private attribute or method: class MyClass: def __init__(self): self._spam = 1 Now what? What are you worried about happening if somebody else gets access to your code, say they download it from Github or something, and use that private attribute? What harm are you trying to prevent? -- Steve
05.05.21 17:18, Shreyan Avigyan пише:
I don't know if it's worth adding private to python modules but what about classes? Private in class is an important feature in OOP that Python lacks (Most OOP languages have private like Java, C++, etc.). I don't know why it was introduced in the first place but it's kind of a convention to have that feature in OOP.
Private fields in Python are no less private than in C++. In C++ you can read private fields: class User { public: string name; private: string password; }; class HackedUser { public: string name; string password; }; password = ((HackedUser*)&user)->password; Of course it is a bad practice. You should not write this in your code, and if your code allows executing arbitrary code in the same address space, your security is already gone. And the same is in Python. You can access private attributes, but usually you should not (except for debugging purpose).
Moreover why does private exist in the first place? If there's a reason then shouldn't it apply to Python as well.
Private exists as a mean of self-limitation. It allows you to separate your data and code on these which are purposed to be used outside of the module or the class, and these which aren't. For your own good you must follow this convention. The same reasons are applied to both C++ and Python. It is more complicated in Java, but its security model is not perfect in any case, and Python prefers the convenience of debugging and simplicity of implementation.
First of all, the reason that private members can be hacked in C++ because of pointers. I don't know Java, so I don't know how it's possible there. But Python doesn't have pointers and getattr, settatr can be adjusted to work with private members. The only way the hack is possible if someone wrote a Extension module in which they would get a PyObject* as a parameter and from there they would manually get the attribute variable using pointer and then return the value. Therefore if private is implemented in Python it would be more secure than private in other OOPs. The benefit of private is actually debugging. If somehow, someone finds that the private member has changed they would immediately know that it was a memory leak or the value was changed through a set method defined in the class (This is one of most important pillar of access modifiers in OOP). Python's private convention most of the time helps but since it doesn't enforce it someone just might introduce a bug in an application by modifying a private member. I just gave an example of security it was not the main point. The main aim of private is to avoid modifying not accessing. I know @property can be used to turn a member variable into a non-modifiable member variable but that also means you can't change it inside the class also. When applying private on functions, I think that's when it becomes private. You're hiding interfaces when you don't need to. In fact hiding interfaces is worse than hacking. Functions can't be hacked. Conclusion ------------ Therefore I actually kind of change my suggestion to introduce a new keyword that avoids modifying member variables from outside the class not the inside. This would preserve @property behavior plus introducing a new kind of behavior. Also this member variables would be inherited.
On Thu, May 6, 2021 at 10:49 PM Shreyan Avigyan <pythonshreyan09@gmail.com> wrote:
The benefit of private is actually debugging. If somehow, someone finds that the private member has changed they would immediately know that it was a memory leak or the value was changed through a set method defined in the class (This is one of most important pillar of access modifiers in OOP). Python's private convention most of the time helps but since it doesn't enforce it someone just might introduce a bug in an application by modifying a private member. I just gave an example of security it was not the main point.
If someone changes a private member of another class and introduces a bug, isn't that on them? The whole point of a single leading underscore is to declare that this isn't part of the public API (namedtuples aside), so you're on your own if you read it - doubly so if you change it. But if someone changes a private member of another class and *fixes* a bug, surely that's a good thing? ChrisA
Most of the time when someone fixes a bug using a private member of another class then it's completely considered private code. Sometimes applications can cause crashes if there's memory leak or modification. Like there's an example where you change the name attribute of a bank account class and then there's a new account file created with that name when we never created or told to create that account file. And as I suggested,
The main aim of private is to avoid modifying not accessing.
On Thu, May 6, 2021 at 10:16 AM Shreyan Avigyan <pythonshreyan09@gmail.com> wrote:
Most of the time when someone fixes a bug using a private member of another class then it's completely considered private code. Sometimes applications can cause crashes if there's memory leak or modification. Like there's an example where you change the name attribute of a bank account class and then there's a new account file created with that name when we never created or told to create that account file. And as I suggested,
The main aim of private is to avoid modifying not accessing.
As everyone knows, in Python we use a single underscore as a signal to other programmers that they should exercise caution if they want to make use of it, and a double underscore (compounded by name-mangling, which I refer to as hyper-private) for an even stronger signal. However, it is part of Python's philosophy to treat programmers as "consenting adults" who can assume risk when warranted. In many of my projects, I have found it desirable to modify some "private" method as the public API is preventing me from achieving a given goal. As an example, see https://github.com/aroberge/friendly/blob/b7be071daaace87dab4aba540f14b9c222... (and L45) where I patch two hyper-private methods of the excellent Rich library to "fix" a limitation of the original API. I absolutely love the fact that I can do this when using Python. I suspect that I am not the only one. Huge -1 for the idea of introducing truly private variables or methods in Python. André Roberge
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/3KI7I4... Code of Conduct: http://python.org/psf/codeofconduct/
Again. See what I suggested. I changed my suggestion from "Private" to "Non-modifiable member variables".
On Thu, May 6, 2021 at 11:22 AM Shreyan Avigyan <pythonshreyan09@gmail.com> wrote: Quoting the entire content: Again. See what I suggested. I changed my suggestion from "Private" to
"Non-modifiable member variables".
I have no idea if this is in response to my own email or someone else as you do not include any context. My own email specifically referred to two instances where I found it necessary to **modify** methods that are indicated as being private as their names begin with double underscores. So, I am strongly opposed to your suggestion. André Roberge
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/LSOF6L... Code of Conduct: http://python.org/psf/codeofconduct/
My own email specifically referred to two instances where I found it necessary to **modify** methods that are indicated as being private as
André Roberge: their names begin with double underscores. So, I am strongly opposed to your suggestion. Actually as I explained in this comment https://mail.python.org/archives/list/python-ideas@python.org/message/426EPA... I'm never suggesting to make member methods non-modifiable. I'm suggesting to add a keyword that can make **member variables** non-modifiable.
I am a strong -1 on this. On Thu, May 6, 2021 at 2:41 PM Shreyan Avigyan <pythonshreyan09@gmail.com> wrote:
My own email specifically referred to two instances where I found it necessary to **modify** methods that are indicated as being private as
André Roberge: their names begin with double underscores. So, I am strongly opposed to your suggestion.
Actually as I explained in this comment https://mail.python.org/archives/list/python-ideas@python.org/message/426EPA... I'm never suggesting to make member methods non-modifiable. I'm suggesting to add a keyword that can make **member variables** non-modifiable. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/XCLFUC... Code of Conduct: http://python.org/psf/codeofconduct/
-- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions.
David Mertz:
I am a strong -1 on this
Thanks for your feedback. But I'm not suggesting introducing a new behavior. The @property already creates read-only member variables. What I'm suggesting is that introducing a new keyword or decorator as required to create a variable that can be modified from within the class but not outside the class.
On Fri, May 7, 2021 at 1:12 AM Shreyan Avigyan <pythonshreyan09@gmail.com> wrote:
David Mertz:
I am a strong -1 on this
Thanks for your feedback. But I'm not suggesting introducing a new behavior. The @property already creates read-only member variables. What I'm suggesting is that introducing a new keyword or decorator as required to create a variable that can be modified from within the class but not outside the class.
That would require some definition of what's "within" and what's "outside" the class, and whatever definition you use, it won't work with all forms of dynamic code. And, as many people have stated, this is not anything that most Python programmers want. ChrisA
Chris:
That would require some definition of what's "within" and what's "outside" the class, and whatever definition you use, it won't work with all forms of dynamic code.
Yes, implementing that will be hard. But the question is I can't quite understand why this is not acceptable by the Python community? Private members may be a disaster but I don't think readonly attributes is. In fact that's what have been implemented for years using @property.
On Fri, May 7, 2021 at 4:04 AM Shreyan Avigyan <pythonshreyan09@gmail.com> wrote:
Chris:
That would require some definition of what's "within" and what's "outside" the class, and whatever definition you use, it won't work with all forms of dynamic code.
Yes, implementing that will be hard. But the question is I can't quite understand why this is not acceptable by the Python community? Private members may be a disaster but I don't think readonly attributes is. In fact that's what have been implemented for years using @property.
I'm not sure about other people, but I have never, not once, used @property as a means of controlling access. So giving me another way to do something that I am not, and don't want to, do... isn't much of an argument. :) ChrisA
I *have* used @property to suggest that variables be read-only. That said, I believe that using the (I believe fairly common) pattern of defining a property as a no-op wrapper around a corresponding "private" (single underscore) attribute tends to be a more pythonic solution to the read-only dilemma. Specifically, that pattern would be defining @property def value(self): return self._value and then just not giving a setter property (or giving a setter that raises NotImplementedError or something similar). In this way, I have made it clear to users that they may use the property and can expect it to be there in the future (it becomes easily documented, and also it offers more guarantees than the underscore "private" attribute). At the same time, I've told the user that it shouldn't be messed with, but any user *could *go in and do what they want with the _value attribute, enabling folks like Andre to make customizations as desired—but at their own risk. Since this pattern is pretty easy to implement, and I think its more pythonic, I'd opt to keep things as they are. On Thu, May 6, 2021 at 11:10 AM Chris Angelico <rosuav@gmail.com> wrote:
On Fri, May 7, 2021 at 4:04 AM Shreyan Avigyan <pythonshreyan09@gmail.com> wrote:
Chris:
That would require some definition of what's "within" and what's "outside" the class, and whatever definition you use, it won't work with all forms of dynamic code.
Yes, implementing that will be hard. But the question is I can't quite
understand why this is not acceptable by the Python community? Private members may be a disaster but I don't think readonly attributes is. In fact that's what have been implemented for years using @property.
I'm not sure about other people, but I have never, not once, used @property as a means of controlling access. So giving me another way to do something that I am not, and don't want to, do... isn't much of an argument. :)
ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/XDTYC7... Code of Conduct: http://python.org/psf/codeofconduct/
Chris:
I'm not sure about other people, but I have never, not once, used @property as a means of controlling access. So giving me another way to do something that I am not, and don't want to, do... isn't much of an argument. :)
Ok. I'm giving you another example that doesn't cause security issues but instead instability. Consider this, class A: def __init__(self, name): self.name = name def create_file(self): with open(name, "a") as file: name.write("something") Now consider this, [1]: x = A("Name") [2]: x.create_file() # Creates a file named "Name" [3]: x.name = "name" [4]: x.create_file() # Creates a file named "name" We never created a object for "name" but still there's a file named "name" out there. This is the work of read-only attribute. They prevent such instability issues.
On 2021-05-06 at 18:34:55 -0000, Shreyan Avigyan <pythonshreyan09@gmail.com> wrote:
Chris:
I'm not sure about other people, but I have never, not once, used @property as a means of controlling access. So giving me another way to do something that I am not, and don't want to, do... isn't much of an argument. :)
Ok. I'm giving you another example that doesn't cause security issues but instead instability. Consider this,
class A: def __init__(self, name): self.name = name def create_file(self): with open(name, "a") as file: name.write("something")
At the risk of summoning Muphry's Law, ITYM file.write.
Now consider this,
[1]: x = A("Name") [2]: x.create_file() # Creates a file named "Name" [3]: x.name = "name" [4]: x.create_file() # Creates a file named "name"
We never created a object for "name" but still there's a file named "name" out there. This is the work of read-only attribute. They prevent such instability issues.
What if I do this: [1]: x = A("arbitrary-string") [2]: x.create_file = lambda: os.unlink(self.name) [3]: x.create_file() # Creates a file named "arbitrary-string" We did create an object for "arbitrary-string" (and invoked it, too) but now there's no such file out there. Should I file a bug report against class A? *wink* This is the work of un-pythonicity. If that attribute is that important, then call it _name instead of name, or make it a property without a setter, or implement setattr as appropriate, or document it as read-only outside of A.__init__. Or even more than one of the above. Then line [3] of your example violates the stated and/or implied API, and is a bug. Python is a Consenting Adults language; if it hurts when you do that, then it's on you not to do that, or at least to report an actual case where it hurt. There are already mechanisms to design, document, and implement mostly read-only data (stipulated: allegedly 100% fool proof solutions never seem to work out that way).
The problem is that whatever programming language it is, there's always a hack to do things. And since it also doesn't make sense to introduce readonly function names (What a terrible nightmare!), maybe readonly variable also doesn't make sense to be added. Anyway, this was quite a discussion. Thanks anyway for considering this. This thread is closed now. Thanking you, With Regards Shreyan Avigyan
On Thu, May 6, 2021, 11:12 AM Shreyan Avigyan <pythonshreyan09@gmail.com> wrote:
David Mertz:
I am a strong -1 on this
Thanks for your feedback. But I'm not suggesting introducing a new behavior. The @property already creates read-only member variables. What I'm suggesting is that introducing a new keyword or decorator as required to create a variable that can be modified from within the class but not outside the class.
Yes, that's exactly what I'm -1 on.
On Thu, May 06, 2021 at 03:09:33PM -0000, Shreyan Avigyan wrote:
What I'm suggesting is that introducing a new keyword or decorator as required to create a variable that can be modified from within the class but not outside the class.
That's an interesting suggestion, but how do you define "within the class"? My guess is that you have used a lot of Java, or at least read a lot of Object Oriented Programming theory that is based on the Java model. In Java "inside the class" is easy: it is a method that is written under the class keyword in the same file as the class. The Java compiler can determine statically which methods belong to the class, which to a parent class or a subclass, and can enforce access rules for "private, public, protected" etc. And the consequence is that, anecdotally, Java programmers spend a lot of time fighting the compiler trying to work around overzealous and unnecessary use of private and protected keywords. They usually do that by using reflection: https://sindhitutorials.com/blog/access-private-variables-and-methods-from-a... Now consider Python. We can, and do, create classes dynamically. For example, we can inject methods into a class at runtime: class Spam(object): pass # Later on def method(self, arg): self._private = arg Spam.method = method The class and the method don't even need to be defined in the same file. We can add methods to instances: from types import MethodType instance = Spam() instance.view = MethodType(lambda self: print(self._private), instance) Both of those are fairly unusual things to do. Maybe less than 1% of classes wouold use those techniques. But a *very* common technique is to use decorators: def decorate(method): def inner(self, arg): # count the number of times the method is called self._counter += 1 return method(self, arg) return inner class Eggs(object): @decorate def method(self, x): do_something_with(x) The important thing here is that at runtime, it is difficult or impossible for the interpreter to tell the difference between a "normal" method which is defined statically inside the class, and a decorated or injected method which came from "outside" the class. It possibly could be done, but only at a significant runtime cost, and it would probably be easy for people to work around. If Python had a "private" keyword that worked as you wanted, it would likely be more of a nuisance than a usual feature, and like Java programmers, we would probably spend too much time working around the overzealous use of the feature. Steve
Steven D'Aprano:
My guess is that you have used a lot of Java, or at least read a lot of Object Oriented Programming theory that is based on the Java model
I've never used Java. I don't even know Java so well. But the Java and C++ model is similar and since I know C/C++ I'm guessing about Java. Anyway this thread is closed.
Ethan:
As you may have noticed, this is a mailing list -- no thread is closed. I've seen posts to three-year-old threads.
Since the discussion has come to a halt and it's decided that we'll continue to use the Pythonic way for private and it's also very hard to implement what I suggested, I think no more discussion should be there regarding this. But if someone is interested then they are always welcome to comment.
I think the only reason to introduce something like `private` is refactoring. If you added a `_variable` and later you decided to expose it, you have to change it to `variable`. This is something that in languages like Java is not necessary, you have only to change the variable from private to public. This sometimes bothered me in Python.
On Sun, May 23, 2021 at 10:30 PM Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
I think the only reason to introduce something like `private` is refactoring. If you added a `_variable` and later you decided to expose it, you have to change it to `variable`. This is something that in languages like Java is not necessary, you have only to change the variable from private to public. This sometimes bothered me in Python.
Since you started with it private, you should be able to solve this with a simple search-and-replace within the class's own definition. Nothing outside the class should be affected. If it's that hard to replace "self._variable" with "self.variable", then you can always create a property to make it available under both names. ChrisA
On Sun, 23 May 2021 at 14:35, Chris Angelico <rosuav@gmail.com> wrote:
On Sun, May 23, 2021 at 10:30 PM Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
I think the only reason to introduce something like `private` is refactoring. If you added a `_variable` and later you decided to expose it, you have to change it to `variable`. This is something that in languages like Java is not necessary, you have only to change the variable from private to public. This sometimes bothered me in Python.
Since you started with it private, you should be able to solve this with a simple search-and-replace within the class's own definition. Nothing outside the class should be affected. If it's that hard to replace "self._variable" with "self.variable"
And, in non-trivial cases, it is :)
then you can always create a property to make it available under both names.
I use property a lot, but I noticed that the majority of programmers do not use them. Also asyncio, written by Guido himself, uses normal getters and setters. Maybe @property slows down the code?
ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/L33KII... Code of Conduct: http://python.org/psf/codeofconduct/
On Sun, May 23, 2021 at 10:42 PM Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
On Sun, 23 May 2021 at 14:35, Chris Angelico <rosuav@gmail.com> wrote:
On Sun, May 23, 2021 at 10:30 PM Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
I think the only reason to introduce something like `private` is refactoring. If you added a `_variable` and later you decided to expose it, you have to change it to `variable`. This is something that in languages like Java is not necessary, you have only to change the variable from private to public. This sometimes bothered me in Python.
Since you started with it private, you should be able to solve this with a simple search-and-replace within the class's own definition. Nothing outside the class should be affected. If it's that hard to replace "self._variable" with "self.variable"
And, in non-trivial cases, it is :)
I'm curious as to what sort of non-trivial cases you have to deal with, where you have something harder than search-and-replace, yet you have to change your mind about whether something's private or public. Are your private members so poorly named that "._variable" comes up a lot? Do you yearn for actual refactoring tools - which do exist?
then you can always create a property to make it available under both names.
I use property a lot, but I noticed that the majority of programmers do not use them. Also asyncio, written by Guido himself, uses normal getters and setters. Maybe @property slows down the code?
Or maybe most Python programmers don't think in terms of private members, and it's easier to just make something public from the start. ChrisA
On Sun, 23 May 2021 at 14:50, Chris Angelico <rosuav@gmail.com> wrote:
On Sun, May 23, 2021 at 10:42 PM Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
On Sun, 23 May 2021 at 14:35, Chris Angelico <rosuav@gmail.com> wrote:
On Sun, May 23, 2021 at 10:30 PM Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
I think the only reason to introduce something like `private` is refactoring. If you added a `_variable` and later you decided to expose it, you have to change it to `variable`. This is something that in languages like Java is not necessary, you have only to change the variable from private to public. This sometimes bothered me in Python.
Since you started with it private, you should be able to solve this with a simple search-and-replace within the class's own definition. Nothing outside the class should be affected. If it's that hard to replace "self._variable" with "self.variable"
And, in non-trivial cases, it is :)
I'm curious as to what sort of non-trivial cases you have to deal with, where you have something harder than search-and-replace
For example, having _variable in another class.
Do you yearn for actual refactoring tools - which do exist?
Renaming tools of IDE do not work in 100% of the cases. For example, if you have _variable in an eval string, it's not replaced.
Or maybe most Python programmers don't think in terms of private members, and it's easier to just make something public from the start.
Naaaa, I see private vars used a lot. It's @property that, it seems to me, is used sporadically.
ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/OT54AL... Code of Conduct: http://python.org/psf/codeofconduct/
On 2021-05-23 at 16:30:35 +0200, Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
On Sun, 23 May 2021 at 14:50, Chris Angelico <rosuav@gmail.com> wrote:
On Sun, May 23, 2021 at 10:42 PM Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
On Sun, 23 May 2021 at 14:35, Chris Angelico <rosuav@gmail.com> wrote:
On Sun, May 23, 2021 at 10:30 PM Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
I think the only reason to introduce something like `private` is refactoring. If you added a `_variable` and later you decided to expose it, you have to change it to `variable`. This is something that in languages like Java is not necessary, you have only to change the variable from private to public. This sometimes bothered me in Python.
Since you started with it private, you should be able to solve this with a simple search-and-replace within the class's own definition. Nothing outside the class should be affected. If it's that hard to replace "self._variable" with "self.variable"
And, in non-trivial cases, it is :)
I'm curious as to what sort of non-trivial cases you have to deal with, where you have something harder than search-and-replace
For example, having _variable in another class.
Then it isn't/wasn't private.
Do you yearn for actual refactoring tools - which do exist?
Renaming tools of IDE do not work in 100% of the cases. For example, if you have _variable in an eval string, it's not replaced.
Another reason not to use eval.
Or maybe most Python programmers don't think in terms of private members, and it's easier to just make something public from the start.
Naaaa, I see private vars used a lot. It's @property that, it seems to me, is used sporadically.
I suspect that I am not "most Python programmers,", but I always think of state (aka members, variables, attributes) as private and internal. If I think I need to reach into an application level object for its state (to read it or to write it), or that I need a getter, let alone a setter, then I'm doing (or have done) somthing else wrong.
On Sun, 23 May 2021 at 17:43, <2QdxY4RzWzUUiLuE@potatochowder.com> wrote:
On 2021-05-23 at 16:30:35 +0200, Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
On Sun, 23 May 2021 at 14:50, Chris Angelico <rosuav@gmail.com> wrote:
On Sun, May 23, 2021 at 10:42 PM Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
On Sun, 23 May 2021 at 14:35, Chris Angelico <rosuav@gmail.com> wrote:
On Sun, May 23, 2021 at 10:30 PM Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
I think the only reason to introduce something like `private` is refactoring. If you added a `_variable` and later you decided to expose it, you have to change it to `variable`. This is something that in languages like Java is not necessary, you have only to change the variable from private to public. This sometimes bothered me in Python.
Since you started with it private, you should be able to solve this with a simple search-and-replace within the class's own definition. Nothing outside the class should be affected. If it's that hard to replace "self._variable" with "self.variable"
And, in non-trivial cases, it is :)
I'm curious as to what sort of non-trivial cases you have to deal with, where you have something harder than search-and-replace
For example, having _variable in another class.
Then it isn't/wasn't private.
I'm not saying _variable of the class A I want to rename is used by the class B. I'm saying that there could exist another class B that has a _variable with the same name. In this case, a simple search and replace all will fail.
Do you yearn for actual refactoring tools - which do exist?
Renaming tools of IDE do not work in 100% of the cases. For example, if you have _variable in an eval string, it's not replaced.
Another reason not to use eval.
I completely agree, but the world is not perfect. Another example is the use of the _variable in a f-string.
Or maybe most Python programmers don't think in terms of private members, and it's easier to just make something public from the start.
Naaaa, I see private vars used a lot. It's @property that, it seems to me, is used sporadically.
I suspect that I am not "most Python programmers,", but I always think of state (aka members, variables, attributes) as private and internal. If I think I need to reach into an application level object for its state (to read it or to write it), or that I need a getter, let alone a setter, then I'm doing (or have done) somthing else wrong. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/MDCRGZ... Code of Conduct: http://python.org/psf/codeofconduct/
On Mon, May 24, 2021 at 3:38 AM Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
Do you yearn for actual refactoring tools - which do exist?
Renaming tools of IDE do not work in 100% of the cases. For example, if you have _variable in an eval string, it's not replaced.
Another reason not to use eval.
I completely agree, but the world is not perfect. Another example is the use of the _variable in a f-string.
f-strings are code, and should be treated as code by any decent refactoring tool. Language features shouldn't be dictated by poor tools, nor by poor codebases. ChrisA
On Sun, 23 May 2021 at 19:50, Chris Angelico <rosuav@gmail.com> wrote:
On Mon, May 24, 2021 at 3:38 AM Marco Sulla <Marco.Sulla.Python@gmail.com> wrote:
Do you yearn for actual refactoring tools - which do exist?
Renaming tools of IDE do not work in 100% of the cases. For example, if you have _variable in an eval string, it's not replaced.
Another reason not to use eval.
I completely agree, but the world is not perfect. Another example is the use of the _variable in a f-string.
f-strings are code, and should be treated as code by any decent refactoring tool.
Ok, but the point is that renaming a variable is always a problem. I check the result of renaming twice even when I use Java with Eclipse. It seems to me that modern languages and programming strategies tend to make refactoring as easy as possible.
Language features shouldn't be dictated by poor tools, nor by poor codebases.
ChrisA _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/6O6NDZ... Code of Conduct: http://python.org/psf/codeofconduct/
Chris :-
Please actually quote some text so we know who you're responding to, what you're talking about, etc.
I'm actually telling to read my comment here https://mail.python.org/archives/list/python-ideas@python.org/message/426EPA.... See the Conclusion section in there.
On 5/6/21 6:26 AM, Shreyan Avigyan wrote:
Chris wrote:
Please actually quote some text so we know who you're responding to, what you're talking about, etc.
I'm actually telling to read my comment here https://mail.python.org/archives/list/python-ideas@python.org/message/426EPA.... See the Conclusion section in there.
I, and many others, read Python List with a mail reader, not a web reader, and have zero desire to start clicking on links to see what you said somewhere else. Copy and paste is your friend. Further, by not quoting others, we have no idea what interpretation of your post you are disagreeing with. So: - quote the portions of posts you are replying to - rewrite, or copy & paste, the portions of your own posts that are relevant - explain why the other respondent's interpretation is not correct - don't bring up security if that's not a primary motivating factor - if your proposal changes, change the subject line to match (or start a new thread) -- ~Ethan~
On Fri, May 7, 2021 at 12:44 AM Ethan Furman <ethan@stoneleaf.us> wrote:
On 5/6/21 6:26 AM, Shreyan Avigyan wrote:
Chris wrote:
Please actually quote some text so we know who you're responding to, what you're talking about, etc.
I'm actually telling to read my comment here https://mail.python.org/archives/list/python-ideas@python.org/message/426EPA.... See the Conclusion section in there.
I, and many others, read Python List with a mail reader, not a web reader, and have zero desire to start clicking on links to see what you said somewhere else. Copy and paste is your friend.
Further, by not quoting others, we have no idea what interpretation of your post you are disagreeing with.
Also: Even if people DO click on the link, there's *still* no indication of who you're disagreeing with. ChrisA
On 2021-05-06 12:46:40, Shreyan Avigyan wrote:
But Python doesn't have pointers and getattr, settatr can be adjusted to work with private members.
Not really, this is explicitly mentioned on the docs. Here's an example:
from dataclasses import dataclass @dataclass(frozen=True) ... class Example: ... field: str ... e1 = Example('string') e1.field 'string' e1.field = 'Something' Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<string>", line 4, in __setattr__ dataclasses.FrozenInstanceError: cannot assign to field 'field' e1.__setattr__('field', 'Something') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<string>", line 4, in __setattr__ dataclasses.FrozenInstanceError: cannot assign to field 'field' object.__setattr__(e1, 'field', 'Something') e1.field 'Something'
"private" member are just a convention, always. No reflection or pointer manipulation is necessary to change them, let alone just reading. "@property", "getattr"/"setattr", "@cached_property" solves all problems that Java introduces with worthless getters and setters.
06.05.21 15:46, Shreyan Avigyan пише:
But Python doesn't have pointers and getattr, settatr can be adjusted to work with private members.
It doesn't work this way. You cannot distinguish the use of getattr() which breaks encapsulation from the one which does not. Inside getattr() you only have an object, the name of the attribute, and possible fallback value. You don't know whether it was called from the method of that object, in which case the access to private attribute should not be restricted, or outside of it. Changing this would require rewriting significant part of the interpreter code, with possible performance loss, introducing new bugs, and making the maintenance more difficult. There would be needed separate storage for private attributes in object (otherwise you could just access them via __dict__). I am not sure even that the goal is achievable in principle, since in Python there is no hard boundary between methods and functions. You can easily add new methods at runtime.
One issue with many common programming languages is that they appear to offer access protections via private/protected etc, but seldom do so securely (`private` modifiers are never, to my knowledge, intended to be a security mechanism). ///////////// Take for example C++: #include <iostream> #include <cstring> class Secrets { private: const std::string password; public: Secrets(const char * password): password(password) {} }; void consumer(Secrets &sec) { auto haxor = (const char *)&sec; for (size_t i=0; i<32; ++i) { std::cout << haxor + i << std::endl; } } int main() { Secrets sec("5exret_Paxxw0rd"); consumer(sec); return 0; } ///////////// It's fairly trivial to bypass the `private` access modified for member variables by reading the memory directly. Even with heap-allocated structures, this is not much harder to do. Likewise for Java: /////// import java.lang.reflect.Field; import java.lang.NoSuchFieldException; class MyClass { private String password; MyClass(String password) { this.password = password; } } class Test { public static void main(String[] a) { MyClass mc = new MyClass("5exret_Paxxw0rd"); try{ Field f = MyClass.class.getDeclaredField("password"); f.setAccessible(true); System.out.println((String)f.get(mc)); } catch(Exception e) { } } } //// Some languages (Java) have optional security managers to try to avoid allowing such things, but they are not always perfect at preventing determined users. My advice would be to design your system to avoid trying to hide information from users who have the ability to run code from within the same address space as the secrets you're trying to protect. Thanks Steve On Wed, May 5, 2021 at 2:44 PM Shreyan Avigyan <pythonshreyan09@gmail.com> wrote:
Private methods, functions and variables are very common in programming languages. But Python doesn't support private. It has conventions for naming so considered private but not private. Most of the time private is never required, what Python provides is more than enough. But the need for private come into place when we're dealing with passphrases and servers. For example consider this code,
class A: def get(): // code to get the password self.password = password
Now consider this,
x = A(); x.get(); x.password
See what just happened? Someone just got the member variable value that the person wasn't supposed to.
I suggest to add private support for functions (module __all__ methods to be more clear), methods and variables (module __all__ variables or class variables).
(I very bad at reading PEPs so I may miss out something critical that's been explained already (sometimes I miss out a feature in a PEP and think about suggesting that feature when it's already there and then I realize "Oh! It's already here in this PEP"). If that's the case then please correct me.)
With Regards, Shreyan Avigyan _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-leave@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/DD2L56... Code of Conduct: http://python.org/psf/codeofconduct/
See my comment here - https://mail.python.org/archives/list/python-ideas@python.org/message/L5LUQD...
Greetings list, This whole thread reminds me of a comment of Miguel Grinberg i remember somewhere when discussing secrets. Some go for .env some for env variables but he suggested focusing the attention on not letting people getting access to the server instead of trying to be clever about hiding secrets. The whole thread echoes sentiments in the same direction. Kind Regards, Abdur-Rahmaan Janhangeer about <https://compileralchemy.github.io/> | blog <https://www.pythonkitchen.com> github <https://github.com/Abdur-RahmaanJ> Mauritius
Abdur-Rahmaan Janhangeer writes:
This whole thread reminds me of a comment of Miguel Grinberg i remember somewhere when discussing secrets. Some go for .env some for env variables but he suggested focusing the attention on not letting people getting access to the server instead of trying to be clever about hiding secrets.
'Support For Private' is really a quite different discussion from 'Defending Secrets', though. It's true that hiding secrets is not a good idea (in particular, on a computer where "dd /dev/sda1 | grep" means everything hidden is hiding in plain sight). On the other hand, if something is an important secret, you should do both: restrict access to the secret for users who have access to the server, and restrict access to the server. (By definition, you can't completely deny access to a server, there has to be some way for clients to get service!) The fundamental issue is a zero-sum game, attacker vs. defender. In Support For Private, however, we're talking about a positive-sum game. Code writers need to communicate with code readers, both human and machine, while avoiding mistakes. In fact, this is almost a purely cooperative game. Another difference is that Secrets Defense is a *repeated* game, like chess: you play, someone wins, you set up the board again and play the same game. Here we have a *stochastic* game, like business, where playing changes the game a bit. The idea of 'private' is to "defend" the API, for example avoiding build up of metastasized technical debt (trying to keep the technical debt in one place, contained within the provider of the API). Here some conflict of interest comes in: consumers of the API want more flexibility or less bureaucracy either for runtime or writetime efficiency. But the discussion is likely to need to be even more nuanced, as the API itself may need to evolve and may represent technical debt. Then it is indeed convenient to be able to change 'private' to '' in one place rather than a global search-replace on '_identifier' to 'identifier'. Of course all of these positive-sum, dynamic considerations enter into the Secrets Defense game too, but that one is much simpler and more encapsulated (yes, I know about GPG and simplicity :-/, but really, it is a much simpler situation). I think it's reasonable to think of repetition of a static game there in a way it simply isn't for 'private' in a programming language. Steve
On Tue, May 25, 2021 at 04:24:39PM +0900, Stephen J. Turnbull wrote:
In Support For Private, however, we're talking about a positive-sum game. Code writers need to communicate with code readers, both human and machine, while avoiding mistakes. In fact, this is almost a purely cooperative game.
It's not really purely cooperative. API consumers always want more than the API designers are willing or able to give. Developers of the API want to hide as much of the system as possible, so they can change it without breaking consumers' code. But consumers want access to as much of the system as possible, so they can do things the developers never thought of when they designed the API, or for performance, or just so they can write one line instead of two. This leads to the well-known conflict between spending resources keeping backwards compatibility, or breaking people's code when you change features. E.g. Microsoft is well-known for the first strategy, up to and including detecting when a specific application is running and emulating the bugs that it relies on. Yes, Microsoft has been known to port bugs from one version of Windows to the next, and I'm not being snarky. Apple, on the other hand, is much more strict and willing to ban or break code that uses features they don't want you to use. "We told you not to do that." In languages that have access controls such as private, protected, final etc, there is a strong tendency for those controls to be over-used by API designers, which then leads consumers of the API to make huge efforts to bypass the controls or work around them. https://stackoverflow.com/questions/27859296/how-to-access-private-class-met... https://www.websparrow.org/java/how-to-access-private-fields-methods-and-con... That's clear evidence of conflict, not cooperation. Now obviously those bypass mechanisms had to be added to the language by the API designers. So even the Java designers recognise that other people over-use private, and so built an entire mechanism (reflection) to allow consumers to bypass it when needed. -- Steve
Steven D'Aprano writes:
On Tue, May 25, 2021 at 04:24:39PM +0900, Stephen J. Turnbull wrote:
It's not really purely cooperative. API consumers always want more than the API designers are willing or able to give.
True enough as stated, but "purely cooperative" doesn't mean "if I optimize, you get the best of all worlds". It means if we communicate, we can coordinate and get a good solution. If we don't communicate, generally we get a bad outcome. The situations you describe don't seem to have a lot of communication in the design of the APIs. There are situations where communication is difficult. Developer A provides an API tuned to Developer B's application, and later Developer C wants A's functionality but wants a different API. Or (as is frequently the case with open source) Developer A is both producer and the initial consumer of the API, and drops the producer on PyPI with an "as-is" notice, frustrating Developer C. Point is, you're right, and lack of communication between API producers and API consumers is why you're right. That is a problem for management; you can't engineer it away, neither with "private, protected, public", nor by avoiding them. Steve
In fact, this is almost a
Greetings, purely cooperative game. This sums it up the difference between secrets and the private use case. I was echoing the sentiments of the thread. The mere putting of private does not ensure complete isolation. It's for developers to understand that they should not directly access the variable. This highlights the approach of Python that if it's for humans, better twerk it for that usecase.
That's clear evidence of conflict, not cooperation.
The intended use is cooperation. If the consumers wish to go into uncharted territory, they take their responsibility! Kind Regards, Abdur-Rahmaan Janhangeer about <https://compileralchemy.github.io/> | blog <https://www.pythonkitchen.com> github <https://github.com/Abdur-RahmaanJ> Mauritius
participants (15)
-
2QdxY4RzWzUUiLuE＠potatochowder.com -
Abdur-Rahmaan Janhangeer -
André Roberge -
Chris Angelico -
David Mertz -
Ethan Furman -
Felipe Rodrigues -
Marco Sulla -
Mitch Negus -
Serhiy Storchaka -
Shreyan Avigyan -
Simão Afonso -
Stephen J. Turnbull -
Stestagg -
Steven D'Aprano