On 6/3/21 4:20 AM, Chris Johns wrote:

Might be out of context here, but IMHO "." shouldn't be assumed to be the current directory anyway.

As someone who has ported python to a system where it isn't, these assumptions tend to cause problems.

That sounds miserable.  What does "." signify on such a system, if not the current directory? //arry/

On 03Jun2021 17:06, Bernat Gabor <jokerjokerer@gmail.com> wrote:

On Thu, Jun 3, 2021 at 5:05 PM Larry Hastings <larry@hastings.org> wrote:

...

On 6/3/21 4:20 AM, Chris Johns wrote: Might be out of context here, but IMHO "." shouldn't be assumed to be the current directory anyway.

As someone who has ported python to a system where it isn't, these assumptions tend to cause problems.

That sounds miserable. What does "." signify on such a system, if not the current directory?

The only other thing that would make sense is a folder in the current working directory named ".", not?

I can't speak for Chris Johns, and we're off topic for this thread. That said, I dislike "." in sys.path, particularly near the front. In fact my own "invoke this module's main" wrapper scripts strip it out. I remain personally of the opinion that it is a security nightmare, making imports _subject_ to the current directory (which might be anywhere, include a directory owned by a hostile user). So i dislike it as a default. I know Robin becker is testing specific behaviour - I don't oppose being _able_ to put "." in sys.path (though I think a concrete absolute path is a saner choice). So for Bernat and Larry: not systems where "." doesn't mean the working directory, but definitely in situations where you want a more secure loading of modules (i.e. only from where I expect, not destabilised by a cd). Cheers, Cameron Simpson <cs@cskk.id.au>

Maybe this? 04732ca993 bpo-43105: Importlib now resolves relative paths when creating module spec objects from file locations (GH-25121) On Thu, Jun 3, 2021 at 9:49 AM Senthil Kumaran <senthil@python.org> wrote:

On 3 Jun 2021, at 09:31, Robin Becker <robin@reportlab.com> wrote:

...

ReportLab has quite a large codebase and I think it would be hard to get a concise test of this behaviour change. I would be glad if this is an expected change a7-->b1 and if the use of '.' in this way has become somehow wrong.

To me, this sounds like an excellent candidate for git bisect to help and figure out which exact commit caused this behavior change.

Was it possible for you to detect that?

Thank you, Senthil _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/CW2SAOGI... Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

On 03/06/2021 18:14, Senthil Kumaran wrote:

On Thu, Jun 03, 2021 at 09:55:57AM -0700, Guido van Rossum wrote:

...

Maybe this?

04732ca993 bpo-43105: Importlib now resolves relative paths when creating module spec objects from file locations (GH-25121)

Likely!. But https://github.com/python/cpython/commit/04732ca993fa077a8b9640cc77fb2f15233... was supposed to a platform specific bug fix.

I didn't a sense that reportlab regression detected was platform specific one. The behavior change is an interesting one though.

The regression may well be a platform issue. I am by no means an expert at building python; I followed a recipe from the ARCH PKGBUILD of some time past so perhaps there are problems from my incompetence.

More interesting thing for me was developers running large test suites between alpha and beta releases, and reporting bugs! :-)

You guys can certainly develop language changes faster than us maintainers can use them; the dead hand of past usage is a real handicap. I have managed a smaller example which breaks for me in arch linux latest. The self contained code is here https://www.reportlab.com/ftp/timport-310b1.py I switched to using importlib.import_module as I think that can now always be used in the versions we support. To generate a test base I used an empty directory and then run (in bash) for d in A B;do rm -rf $d; mkdir $d;echo "print('+++++ in module %s' % __name__)" > $d/$(echo $d | tr [:upper:] [:lower:]).py; done or $ for d in A B;do

rm -rf $d mkdir $d echo "print('+++++ in module %s' % __name__)" \ > $d/$(echo $d | tr [:upper:] [:lower:]).py done

to create two simple modules in dirs A & B. If I run python timport-310b1.py in 3.10.0a7 I occasionally see a problem, but in 3.10.0b1 or b2 I see the problem every time; the TB ends like this

print(recursiveImport(mn, debug=1)) File "/home/robin/devel/reportlab/REPOS/timpbug/tests/timport-310b1.py", line 36, in recursiveImport return importlib.import_module(modulename) File "/home/robin/devel/reportlab/.py310/lib/python3.10/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "<frozen importlib._bootstrap>", line 1050, in _gcd_import File "<frozen importlib._bootstrap>", line 1027, in _find_and_load File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked ModuleNotFoundError: No module named 'a'

The occasional problem with 3.10.0a7 make me think some random element is at work perhaps hashing stuff. -- Robin Becker

On 6/3/2021 7:42 PM, Senthil Kumaran wrote:

On Thu, Jun 03, 2021 at 07:08:06PM +0100, Robin Becker wrote:

...

The regression may well be a platform issue. I am by no means an expert at building python; I followed a recipe from the ARCH PKGBUILD of some time

I meant the change in the diff we were suspecting was supposed to be "Windows" specific. But interesting that it was observed in ARCH. The non-windows piece of changes were probably responsible

While we were investigating the reliably broken behaviour, we figured that related behaviour was *unreliably* broken on all platforms. Specifically, if you imported a module through a relative path (resolved to CWD), changed the CWD, cleared the module cache, and reimported, you would still get the original module, contrary to the intent of getting the newly resolved module. ("Correct" code would have also cleared the import path caches, not just sys.modules.) This was because the module info was cached using the relative path, and so would be used later even though its absolute path had changed. Someone reported this change in 3.8 and we decided to revert it in existing released, because the bugfix was for an obscure enough situation that it really wasn't worth breaking any existing versions. For unreleased versions, it's better to fix it so that imports will be more reliable in the future. Based on Robin's original post, it looks like this is exactly the scenario we wanted to fix, and it's probably exposed an underlying issue in that test suite. Unfortunately, import caching has changed a lot since Python 2.7, so it's very unlikely that one approach will be able to do reliable reloading across all of those versions. Your best bet since 3.3 is to use the importlib module's methods (but your *best* bet overall is to run separate test suites in their own process so you can avoid the CWD changes). Cheers, Steve

On Sat., Jun. 5, 2021, 07:38 Robin Becker, <robin@reportlab.com> wrote:

On 04/06/2021 19:29, Steve Dower wrote: ...........

...

While we were investigating the reliably broken behaviour, we figured

that related behaviour was

...

*unreliably* broken on all platforms.

Specifically, if you imported a module through a relative path (resolved to CWD), changed the CWD, cleared the module cache, and reimported, you would still get the original module, contrary to the intent of getting the newly resolved module. ("Correct" code would have also cleared the import path caches, not just sys.modules.) This was because the module info was cached using the relative path, and so would be used later even though its absolute path had changed.

Someone reported this change in 3.8 and we decided to revert it in existing released, because the bugfix was for an obscure enough situation that it really wasn't worth breaking any existing versions. For unreleased versions, it's better to fix it so that imports will be more reliable in the future.

Unfortunately we are using these relative imports to implement plugins for an xml language processor so this is likely not just an accident of ordering the failing tests. I agree we have to get a plugin module a bit more directly using the import internals. I still think we might need to control sys.path as presumably the plugin code could have imports of parallel modules.

I'm beginning to understand that messing about with sys.path/modules/meta_path etc etc is a lot harder than it used to be.

How so? The key details have not changed since Python 3.4 when module specs were introduced. And meta path, etc. are 2.7 concepts.

Incidentally I find that I am able to import a namespace module directly; this code doesn't crash (at least in python >= 3.6

################################################### #timpA.py don't run with any meaningful directory A import importlib def main(): module = importlib.import_module('A') print('module=%r' % module)

if __name__=='__main__': main() ###################################################

in python 3.6-3.9 this prints out "module=<module 'A' (namespace)>" in 3.10.0b1 it prints "module=<module 'A' (<_frozen_importlib_external._NamespaceLoader object at 0x7f1c8a22fe20>)>". I thought I had to use pkgutils or pkg_resources to mess with these new-fangled packages.

Nope, that has never been required since the concept of namespace packages were introduced to import.

Likewise I find that the python statement 'import A' also doesn't fail and A is a namespace.

Yep, that's expected. I feel quite stupid now :(

As soon as you step outside the norm there are a lot of details to understand; import is very much following the "Complex is better than complicated" zen. So no need to feel stupid, there's a lot to keep track of when you start to dive into the details. -Brett --

Robin Becker _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KMJAQUGS... Code of Conduct: http://python.org/psf/codeofconduct/