problem with Distributed File System Replication and Namespacing and different versions of Python 3
Hi, I have a question about a problem we are facing in different Python versions, like Python 3.7.7, Python 3.8, Python 3.6 and Python 3.9. This apparently doesnt happen in Python 3.10.2 and we were wondering how we could achieve the same behaviours without going deep into the nuts and bolts of the python import system. We have python libraries that are distributed in a networked environment that when imported for the first time they are correctly imported but something happens in the meantime that prevents it from loading anymore. This networked paths are domain paths that are bound to a final "real" path in a specific file server from a specific site in the world. What happens is, when injecting into the sys.path the domain names it doesnt import but when injecting into the sys.path the "real" file server path it works, generally speaking. We have been facing this issue in such different python 3 versions and i was wondering what makes this work in Python 3.10.2. Apparently, in my opinion it is a mix of two things involved: one, something the DFS-R DFS-N does at a very low level, that python 3 versions dont like when importing and the other one, the fact that is a network path. Can anybody shed some light about the improvements regarding this? We will be tied to all these python versions for a very long time and we need to give solution to this. One of the possibilities would be to distribute the libraries into C:/ local disc, this eliminating the network path variable. But it would be nice to know what is causing this in Python < 3.10.2. Thanks in advance.
Not sure if this is relevant, but I know they updated the openssl version in Python 3.10. I had the opposite problem trying to use Python 3.10 on CentOS 7 which has the old openSSL. I had to back off to 3.9. maybe running over the network invoked openSSL at some point and that's where the disconnect occurs? I would check which openSSL your host os uses. On Thu, Oct 20, 2022, 2:36 PM <rainonthescarecrowhumanwheels@gmail.com> wrote:
Hi, I have a question about a problem we are facing in different Python versions, like Python 3.7.7, Python 3.8, Python 3.6 and Python 3.9. This apparently doesnt happen in Python 3.10.2 and we were wondering how we could achieve the same behaviours without going deep into the nuts and bolts of the python import system.
We have python libraries that are distributed in a networked environment that when imported for the first time they are correctly imported but something happens in the meantime that prevents it from loading anymore. This networked paths are domain paths that are bound to a final "real" path in a specific file server from a specific site in the world.
What happens is, when injecting into the sys.path the domain names it doesnt import but when injecting into the sys.path the "real" file server path it works, generally speaking. We have been facing this issue in such different python 3 versions and i was wondering what makes this work in Python 3.10.2.
Apparently, in my opinion it is a mix of two things involved: one, something the DFS-R DFS-N does at a very low level, that python 3 versions dont like when importing and the other one, the fact that is a network path.
Can anybody shed some light about the improvements regarding this?
We will be tied to all these python versions for a very long time and we need to give solution to this. One of the possibilities would be to distribute the libraries into C:/ local disc, this eliminating the network path variable. But it would be nice to know what is causing this in Python < 3.10.2.
Thanks in advance. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5EJ7FAWQ... Code of Conduct: http://python.org/psf/codeofconduct/
Hi, not sure about this neither. I was hoping someone with more experience deploying pythons and executing them over distributed file systems would shed a light why this happens. I'm sorry but to be honest i dont see the correlation between OpenSSL and DFS-R/DFS-N, maybe there is one but im a bit lost here. Thanks.
I guess my simplified question would be: ¿how does Python 3 versions manage Distributed File Systems and in particular Windows' DFS? Is it supported by the "import system" in Python? What differences are there?
On Fri, Oct 21, 2022 at 11:11 AM Juan Cristóbal Quesada < rainonthescarecrowhumanwheels@gmail.com> wrote:
I guess my simplified question would be:
¿how does Python 3 versions manage Distributed File Systems and in particular Windows' DFS?
In no specific way. We typically ask the OS to handle any file system details.
Is it supported by the "import system" in Python? What differences are there?
Not explicitly. Everything is done via file paths and things in the os module (technically the posix or nt module depending on what OS you are running on). -Brett
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/HS7KIAFP... Code of Conduct: http://python.org/psf/codeofconduct/
Import is implementing in Python itself via importlib, so you could try walking through the code with a debugger to see where the difference may be (I personally can't think of anything obvious that changed in importlib between 3.9 and 3.10 that would affect this). On Thu, Oct 20, 2022 at 11:36 AM <rainonthescarecrowhumanwheels@gmail.com> wrote:
Hi, I have a question about a problem we are facing in different Python versions, like Python 3.7.7, Python 3.8, Python 3.6 and Python 3.9. This apparently doesnt happen in Python 3.10.2 and we were wondering how we could achieve the same behaviours without going deep into the nuts and bolts of the python import system.
We have python libraries that are distributed in a networked environment that when imported for the first time they are correctly imported but something happens in the meantime that prevents it from loading anymore. This networked paths are domain paths that are bound to a final "real" path in a specific file server from a specific site in the world.
What happens is, when injecting into the sys.path the domain names it doesnt import but when injecting into the sys.path the "real" file server path it works, generally speaking. We have been facing this issue in such different python 3 versions and i was wondering what makes this work in Python 3.10.2.
Apparently, in my opinion it is a mix of two things involved: one, something the DFS-R DFS-N does at a very low level, that python 3 versions dont like when importing and the other one, the fact that is a network path.
Can anybody shed some light about the improvements regarding this?
We will be tied to all these python versions for a very long time and we need to give solution to this. One of the possibilities would be to distribute the libraries into C:/ local disc, this eliminating the network path variable. But it would be nice to know what is causing this in Python < 3.10.2.
Thanks in advance. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/5EJ7FAWQ... Code of Conduct: http://python.org/psf/codeofconduct/
Thanks, yes, that was my last resort: inspecting the code through debugging the import system. Was hoping to get some light before that ;).
On 10/20/2022 1:07 PM, rainonthescarecrowhumanwheels@gmail.com wrote:
What happens is, when injecting into the sys.path the domain names it doesnt import but when injecting into the sys.path the "real" file server path it works, generally speaking. We have been facing this issue in such different python 3 versions and i was wondering what makes this work in Python 3.10.2.
We added some path fixes into importlib in 3.10 that will result in better path resolution during import on Windows, particularly when resolving partial paths. You'll find them in Lib/importlib/_bootstrap_external.py and Modules/posixmodule.c (they're not real obvious - you'll be best to diff those files). You're unlikely to easily backport them to earlier versions, so your current workaround is probably best. If the problems are due to *writing* bytecode files, you might also be able to use PYTHONPYCACHEPREFIX to reference a local directory and avoid issues that way. Cheers, Steve
Hi Steve, thanks! Will definitely have a look at it as soon as i can. Many thanks to all of you that replied. It was my first post in such python mailing lists and wasnt sure how accurate of a response i could have. You never know how active the mailing lists/forums are. Best Regards, JC
I don't remember it being mentioned, but much of the traffic recently migrated from this list to https://discuss.python.org/c/core-dev/23, which you may wish to keep in touch with. Kind regards, Steve On Tue, Oct 25, 2022 at 7:53 AM Juan Cristóbal Quesada < rainonthescarecrowhumanwheels@gmail.com> wrote:
Hi Steve, thanks! Will definitely have a look at it as soon as i can.
Many thanks to all of you that replied. It was my first post in such python mailing lists and wasnt sure how accurate of a response i could have. You never know how active the mailing lists/forums are.
Best Regards, JC _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/EJK5AFAT... Code of Conduct: http://python.org/psf/codeofconduct/
participants (6)
-
Brett Cannon -
Juan Cristóbal Quesada -
Lou King -
rainonthescarecrowhumanwheels＠gmail.com
-
Steve Dower -
Steve Holden