On 23/08/2017 20:49, Chris Angelico wrote:

On Thu, Aug 24, 2017 at 3:37 AM, John Torakis wrote:

...

On 23/08/2017 20:36, John Torakis wrote:

...

Yeah, I am a security researcher, I am keen on backdoor programming and staging and all that! It is my official job and research topic! I go to the office and code such stuff! I am not a blackhat, nor a security enthusiast, it is my job.

First of all, let's all agree that if someone can run Python code in your computer you are 100% hacked! It is irrelevant if "httpimport" is a core python feature or not in that case.

Now, I agree that this can be exploited if used under plain HTTP, it is a MiTM -> Remote code execution case. I admit that this is not bright. But I mention that this can be used in testing.

On the topic of HTTPS, man-in-the-middle is not possible without previous Trusted Certificate compromise. Github can be trusted 100% percent for example. A certificate check has to take place in the HTTPS remote loading for sure!

Right, but that just pushes the problem one level further out: you need to have a 100% dependable certificate chain. And that means absolutely completely trusting all of your root certificates, and it also means either not needing to add any _more_ root certificates, or being able to configure the cert store. As we've seen elsewhere, this is nontrivial. The centralized PKI as we know it is a pain altogether. Please let me reference this XKCD strip here: https://xkcd.com/1200/ Running code to your computer is among the low Impact things that can happen to you if you have a compromised certificate store. Trust me on that! In other words, if you can't trust your certificate store now, and you are afraid of Remote code execution through HTTPS, stop using pip altogether: https://github.com/pypa/pip/issues/1168

And most Package Managers anyway..

...

...

When I said a "core feature" I meant that the "httpimport" module would deliver with the core modules. Not that the Finder/Loader has to be in the list of Finders/Loaders that are used by default! For god sake, I wouldn't like my PC to start probing for modules just because I mistyped an import line! Glad we agree about that! I have seen people wanting all sorts of things to become core features (usually for the sake of interactive work), and a lot of it is MUCH better handled as a non-core feature.

Though a lot of what you're saying here - especially this:

...

...

I know that pip works nicely, especially when paired with virtual environments, but ad-hoc importing is another another thing. It isn't meant for delivering real projects. Just for testing modules without the need to download them, maybe install them, and all. could be equally well handled by pip-installing httpimport itself, and using that to bootstrap your testing procedures. Unless, of course, you're wanting to httpimport httpimport, in which case you're going to run into bootstrapping problems whichever way you do it :) I will never open an issue for 'httpimporting the httpimport itself' It is a promise!

I think we're on the same page here, but it definitely needs some more text in the README to explain this - particularly how this is not a replacement for pip. For example, my first thought on seeing this was "wow, that's going to be abysmally slow unless it has a cache", but the answer to that is: if you need a cache, you probably should be using pip to install things properly.

Still -1 on this becoming a stdlib package, as there's nothing I've yet seen that can't be done as a third-party package. But it's less scary than I thought it was :) The reason I though that this could serve greatly as an stdlib package is that it will broaden the horizon for importing arbitrary stuff just to see if they are working as expected. "Testing" is the world!

ChrisA _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

This isn't ever going to be a standard feature. It's available as a third-party package and that's fine. I'd like to add a historic note -- this was first proposed around 1995 by Michael McLay. (Sorry, I don't have an email sitting around, but I'm sure he brought this up at or around the first Python workshop at NIST in 1995 -- I was his guest at NIST for several months at the time.) -- --Guido van Rossum (python.org/~guido)

For security reasons. AFAIK HTTPS wasn't even invented at the time. On Wed, Aug 23, 2017 at 11:44 AM, John Torakis wrote:

On 23/08/2017 21:41, Guido van Rossum wrote:

This isn't ever going to be a standard feature. It's available as a third-party package and that's fine.

I'd like to add a historic note -- this was first proposed around 1995 by Michael McLay. (Sorry, I don't have an email sitting around, but I'm sure he brought this up at or around the first Python workshop at NIST in 1995 -- I was his guest at NIST for several months at the time.)

Woah! I was 2 years old at that time! Little did I know! Can I ask why it got rejected the first time?

-- --Guido van Rossum (python.org/~guido http://python.org/%7Eguido)

_______________________________________________ Python-ideas mailing listPython-ideas@python.orghttps://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido)

On Thu, Aug 24, 2017 at 5:04 AM, John Torakis wrote:

Dark times...

So is it a "case closed", or is there any improvement that will make it worth it to be an stdlib module?

I mean, times have changed from 1995, and I am not referring to HTTPS invention. This is the reason that makes httpimport just tolerable security-wise.

I'm talking about the need to rapidly test public code. I insist that testing code available on Github (or other repos), without the venv/clone/install hassle is a major improvement in my (and most sec researchers' I know) Python workflow. It makes REPL prototyping million times smoother. We all have created small scripts that auto load modules from URLs anyway. That's why I thought that this modules falls under the second category of 20.2.1 in https://docs.python.org/devguide/stdlibchanges.html (I did my homework before getting to mail in this list).

So, if there is something that would make this module acceptable for stdlib, please let me know! I'd more than happily reform it and make it comply with Python stdlib requirements.

Why can't people just "pip install httpimport" to make use of it? Why does it need to be in the stdlib? ChrisA

John Torakis writes:

But, as it seems like it is a very big feature (to me at least),

From the point of view of the blue team, checking for mere presence of httpimport in the environment is indicative of danger if it's

And "pip install httpimport" seems like it is a very small burden (to me at least). I agree with Paul Moore. Putting this in the stdlib seems both unnecessary, given pip, and an attractive nuisance for naive users. pip-able, useless if it's in the stdlib. With respect to "it just makes exec(urlopen()) easier", any code must be audited for application of exec() to user input anyway, regardless of whether it fetches stuff off the Internet. Adding httpimport use to the checklist adds a little bit of complexity to *every* security check, and a fair amount of danger in security-oblivious environments such as many university labs, and I would imagine many corporate development groups as well. YMMV, but from the point of view of the larger, security-conscious organization, I would say -1. It's an attractive nuisance unless you're a security person, and then pip is not a big deal. Steve

On 23/08/2017 21:24, Paul Moore wrote:

...

Still -1 on this becoming a stdlib package, as there's nothing I've yet seen that can't be done as a third-party package. But it's less scary than I thought it was :) IMO, this would make a great 3rd party package (I note that it's not yet published on PyPI). It's possible that it would end up being extremely popular, and recognised as sufficiently secure - at which

On 23 August 2017 at 18:49, Chris Angelico wrote: point it may be worth considering for core inclusion. But it's also possible that it remains niche, and/or people aren't willing to take the security risks that it implies, in which case it's still useful to those who do like it. PyPI upload is scheduled when some more testing and commenting takes place. One aspect that hasn't been mentioned yet - as a 3rd party module, the user (or the organisation's security team) can control whether or not the ability to import over the web is available by controlling whether the module is allowed to be installed - whereas with a core module, it's there, like it or not, and *all* Python code has to be audited on the assumption that it might be used. True! But you can urlopen()->exec() anything out there anyway! A ">>>" prompt is all you need. I could easily imagine cases where the httpimport module was allowed on development machines and CI servers, but forbidden on production (and pre-production) systems. That option simply isn't available if the feature is in the core. I agree that there are circumstances that this module should not be used (regardless of security implications). In a released product for example. Depending on the UP-ness of a remote repository (e.g github), not to even mention the API backward-compatibility of an upstream package, is just **BAD** for a ready-released-deliverable product! This is why we have virtual environments!

But it remains an option to use it or not! I, for example, find myself REPLing more than scripting. When REPLing for something you plan to implement sometime-somehow, this module is really what you need! But when I finally create a script, I won't disable its offline functionality just to use httpimport. That would be suicidal! When I finally come with a working thing I will finally land the used packages to disk and to a virtual environment. My argument is that this module will add greatly to the Python's ad-hoc testing capabilities! I find it elegant for such feature to be in the stdlib of a language. I don't doubt that it can survive as a 3rd party module, though.

Paul _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On Thu, Aug 24, 2017 at 4:04 AM, Bruce Leban wrote:

On Wed, Aug 23, 2017 at 10:37 AM, John Torakis wrote:

...

Github can be trusted 100% percent for example.

This isn't even remotely close to true. While I'd agree with the statement that the SSL cert on github is reasonably trustworthy, the *content* on github is NOT trustworthy and that's where the security risk is.

I agree that this is a useful feature and there is no way it should be on by default. The right way IMHO to do this is to have a command line option something like this:

python --http-import somelib=https://github.com/someuser/somelib

If you read his README, it's pretty explicit about URLs; the risk is that "https://github.com/someuser/somelib" can be intercepted, not that "someuser" is malicious. If you're worried about the latter, don't use httpimport. ChrisA

On Wed, Aug 23, 2017 at 11:11 AM, Chris Angelico wrote:

If you read his README, it's pretty explicit about URLs; the risk is that "https://github.com/someuser/somelib" can be intercepted, not that "someuser" is malicious. If you're worried about the latter, don't use httpimport.

I don't see the word "security" or "risk" in the readme. The risk is not just that someuser is malicious but the risk that they, their github credentials or their code have been compromised. The reason that if this feature were to be implemented, I would want it outside the source code (command line option) is that that puts the control in the hands of the person running the code. This is appropriate for the stated scenarios. There's no possibility of a hidden live github dependency. --- Bruce

On Thu, Aug 24, 2017 at 12:13 PM, Stephen J. Turnbull wrote:

Chris Angelico writes:

...

If you're worried about the latter, don't use httpimport.

I guarantee you that in my (university) environment, if httpimport is in the stdlib, its use will be rampant (and not just by students, but by security-oblivious faculty). I want to be able to walk up to a student, say "may I?" and type "python -m httpimport" to determine if that particular risky behavior is a worry. Because *I'm* liable for my students' PCs' behavior on the network.

Personally speaking, +1 on PyPI, -100 on stdlib.

Agreed, and a VERY good reason for this to be an explicitly-installed package. By its nature, it won't be a dependency of other packages, so keeping it out of the stdlib pretty much guarantees that it'll only be available if it's been called for by name. ChrisA