Transfering modules
Hello, I have a question that I haven't been able to find online or in the python documentation. While trying to install a module I have encountered significant trouble, stalling my current experiment. I recently installed python 3.6.2, and I want to know how to transfer the numerous modules I have on 3.6.1 to the newest version, usable by an interpreter. Since I was not able to transfer packages to the newer version, I tried to install my module to the working python 3.6.1 version, but similarly, could not find any documentation about how to do this. I also tried to completely uninstall the python 3.6.2 build, but after running sudo-remove commands, and then testing by booting python3, the build running is still 3.6.2, even after removing the framework etc. I am sincerely confused by this and it has unfortunately brought my experiment to a standstill. If you could offer any illumination on this, I would be extremely grateful. I truly thank you in advance for any help you can provide. Best regards, Ian Hartley
On 18 August 2017 at 12:17, Ian Hartley <iahartle@ucsd.edu> wrote:
Hello,
I have a question that I haven't been able to find online or in the python documentation. While trying to install a module I have encountered significant trouble, stalling my current experiment.
I recently installed python 3.6.2, and I want to know how to transfer the numerous modules I have on 3.6.1 to the newest version, usable by an interpreter.
Without knowing more about your system (operating system, where you installed Python from, how you're installing Python dependencies), it's hard to say what might be going wrong, since most mechanisms for upgrading Python will treat 3.6.2 as a direct replacement for 3.6.1, and hence you shouldn't have to do anything at all to start using Python 3.6.2 instead (since they're part of the same feature release stream, they'll both use `3.6` to qualify the directories they access, and hence should be able to see the same set of importable modules). However, from your comments, it sounds like you managed to do a parallel install instead, in which case the answers would be: 1. Don't do that, as you'll make life more difficult for yourself by defeating the main point of offering in-place maintenance updates 2. If you really need to do it, *and* you're using a virtual environment to manage your dependencies, then you can technically change which Python an existing virtual environment is using by fiddling with the symlinks (but I wouldn't recommend it) 3. If you're *not* using a virtual environment to manage your dependencies, then running `python3 -m site` with each version will tell you where their respective site-packages are, so you can copy all the files from your 3.6.1 installation to your new 3.6.2 installation (although again, I wouldn't really recommend it) The ideal case would be having all your dependencies listed in a requirements.txt file, so setting up a new virtual environment would just be a matter of activating it and then running "pip install -r requirements.txt". (Or, equivalently, define a conda environment file and set up your experimental system with "conda env create": https://conda.io/docs/using/envs.html#create-environment-file-by-hand) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
You could potentially also run `/path/to/python3.6.2/bin/python3 -m pip install $(/path/to/python3.6.1/bin/python3 -m pip freeze)`, naturally replacing path with the correct path. pip freeze lists the versions of things you have installed. HTH, Wayne On Fri, Aug 18, 2017, 3:03 AM Nick Coghlan <ncoghlan@gmail.com> wrote:
Hello,
I have a question that I haven't been able to find online or in the
On 18 August 2017 at 12:17, Ian Hartley <iahartle@ucsd.edu> wrote: python
documentation. While trying to install a module I have encountered significant trouble, stalling my current experiment.
I recently installed python 3.6.2, and I want to know how to transfer the numerous modules I have on 3.6.1 to the newest version, usable by an interpreter.
Without knowing more about your system (operating system, where you installed Python from, how you're installing Python dependencies), it's hard to say what might be going wrong, since most mechanisms for upgrading Python will treat 3.6.2 as a direct replacement for 3.6.1, and hence you shouldn't have to do anything at all to start using Python 3.6.2 instead (since they're part of the same feature release stream, they'll both use `3.6` to qualify the directories they access, and hence should be able to see the same set of importable modules).
However, from your comments, it sounds like you managed to do a parallel install instead, in which case the answers would be:
1. Don't do that, as you'll make life more difficult for yourself by defeating the main point of offering in-place maintenance updates 2. If you really need to do it, *and* you're using a virtual environment to manage your dependencies, then you can technically change which Python an existing virtual environment is using by fiddling with the symlinks (but I wouldn't recommend it) 3. If you're *not* using a virtual environment to manage your dependencies, then running `python3 -m site` with each version will tell you where their respective site-packages are, so you can copy all the files from your 3.6.1 installation to your new 3.6.2 installation (although again, I wouldn't really recommend it)
The ideal case would be having all your dependencies listed in a requirements.txt file, so setting up a new virtual environment would just be a matter of activating it and then running "pip install -r requirements.txt". (Or, equivalently, define a conda environment file and set up your experimental system with "conda env create": https://conda.io/docs/using/envs.html#create-environment-file-by-hand)
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Thanks all, I was able to copy the files from the site-packages, and transfer to 3.6.2. For some reason, even though I've transfered all the modules, only some are appearing. Very strange. Either way I think it should work for now... Let me know if you know why only some of the packages would be showing up. Thanks for all your help. Best, Ian On Fri, Aug 18, 2017 at 4:57 AM, Wayne Werner <waynejwerner@gmail.com> wrote:
You could potentially also run `/path/to/python3.6.2/bin/python3 -m pip install $(/path/to/python3.6.1/bin/python3 -m pip freeze)`, naturally replacing path with the correct path. pip freeze lists the versions of things you have installed.
HTH, Wayne
On Fri, Aug 18, 2017, 3:03 AM Nick Coghlan <ncoghlan@gmail.com> wrote:
Hello,
I have a question that I haven't been able to find online or in the
documentation. While trying to install a module I have encountered significant trouble, stalling my current experiment.
I recently installed python 3.6.2, and I want to know how to transfer
On 18 August 2017 at 12:17, Ian Hartley <iahartle@ucsd.edu> wrote: python the
numerous modules I have on 3.6.1 to the newest version, usable by an interpreter.
Without knowing more about your system (operating system, where you installed Python from, how you're installing Python dependencies), it's hard to say what might be going wrong, since most mechanisms for upgrading Python will treat 3.6.2 as a direct replacement for 3.6.1, and hence you shouldn't have to do anything at all to start using Python 3.6.2 instead (since they're part of the same feature release stream, they'll both use `3.6` to qualify the directories they access, and hence should be able to see the same set of importable modules).
However, from your comments, it sounds like you managed to do a parallel install instead, in which case the answers would be:
1. Don't do that, as you'll make life more difficult for yourself by defeating the main point of offering in-place maintenance updates 2. If you really need to do it, *and* you're using a virtual environment to manage your dependencies, then you can technically change which Python an existing virtual environment is using by fiddling with the symlinks (but I wouldn't recommend it) 3. If you're *not* using a virtual environment to manage your dependencies, then running `python3 -m site` with each version will tell you where their respective site-packages are, so you can copy all the files from your 3.6.1 installation to your new 3.6.2 installation (although again, I wouldn't really recommend it)
The ideal case would be having all your dependencies listed in a requirements.txt file, so setting up a new virtual environment would just be a matter of activating it and then running "pip install -r requirements.txt". (Or, equivalently, define a conda environment file and set up your experimental system with "conda env create": https://conda.io/docs/using/envs.html#create-environment-file-by-hand)
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 19 August 2017 at 04:39, Ian Hartley <iahartle@ucsd.edu> wrote:
Thanks all,
I was able to copy the files from the site-packages, and transfer to 3.6.2. For some reason, even though I've transfered all the modules, only some are appearing. Very strange. Either way I think it should work for now...
Let me know if you know why only some of the packages would be showing up.
I'm still unclear on how you even managed to get both 3.6.1 and 3.6.2 installed on the same system (since 3.6.2 would overwrite 3.6.1 entirely in most circumstances), and without knowing that it's hard to guess what might be going wrong in transferring packages between the two :) Would you mind posting the output of "python3 -m site" for both installations? Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, Aug 18, 2017 at 1:03 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 18 August 2017 at 12:17, Ian Hartley <iahartle@ucsd.edu> wrote:
I recently installed python 3.6.2, and I want to know how to transfer the numerous modules I have on 3.6.1 to the newest version, usable by an interpreter.
Without knowing more about your system (operating system, where you installed Python from, how you're installing Python dependencies), it's hard to say what might be going wrong, since most mechanisms for upgrading Python will treat 3.6.2 as a direct replacement for 3.6.1, and hence you shouldn't have to do anything at all to start using Python 3.6.2 instead (since they're part of the same feature release stream, they'll both use `3.6` to qualify the directories they access, and hence should be able to see the same set of importable modules).
I would just like to point out that the normal workflow for pyenv [1], which seems very popular (and which I use), is to do parallel installs rather than upgrades. The OP didn't actually say "upgrade," so it's possible they're using a tool like pyenv. I believe the OP's original questions would also apply to users of pyenv -- i.e. having a simple way to "copy" installed dependencies from one pyenv install to a new one. In practice, I myself don't have this issue since I usually install from requirements.txt, as you suggested, Nick. --Chris [1] https://github.com/pyenv/pyenv
However, from your comments, it sounds like you managed to do a parallel install instead, in which case the answers would be:
1. Don't do that, as you'll make life more difficult for yourself by defeating the main point of offering in-place maintenance updates 2. If you really need to do it, *and* you're using a virtual environment to manage your dependencies, then you can technically change which Python an existing virtual environment is using by fiddling with the symlinks (but I wouldn't recommend it) 3. If you're *not* using a virtual environment to manage your dependencies, then running `python3 -m site` with each version will tell you where their respective site-packages are, so you can copy all the files from your 3.6.1 installation to your new 3.6.2 installation (although again, I wouldn't really recommend it)
The ideal case would be having all your dependencies listed in a requirements.txt file, so setting up a new virtual environment would just be a matter of activating it and then running "pip install -r requirements.txt". (Or, equivalently, define a conda environment file and set up your experimental system with "conda env create": https://conda.io/docs/using/envs.html#create-environment-file-by-hand)
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 19 August 2017 at 14:03, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
I would just like to point out that the normal workflow for pyenv [1], which seems very popular (and which I use), is to do parallel installs rather than upgrades. The OP didn't actually say "upgrade," so it's possible they're using a tool like pyenv.
Right, it was the references to sudo-remove, rebooting, and python3 that indicated to me we were talking system wide operations, rather than any of the environment management tools like pew, pyenv, conda, etc. That's why I'm somewhat puzzled, since even a raw "make altinstall" will overwrite an existing installation of that version. The most likely candidate for a system level kinda-sorta parallel installation would be via the distinction between /usr/local/* and /usr/*, and if it's a Debian or Ubuntu system, that would result in the actual system version being able to see components in dist-packages that would be invisible to an unpatched upstream build, each version having its own site-packages directory, but both of them still sharing a common user site-packages directory. Hence my interest in the output of "<path-to-python> -m site" for the affected versions: seeing the default contents of sys.path will likely answer the question of what's going on definitively, rather than having to guess based on incomplete information. I believe the OP's original
questions would also apply to users of pyenv -- i.e. having a simple way to "copy" installed dependencies from one pyenv install to a new one.
As Wayne noted, generating a requirements.txt file with `pip freeze` and then reinstalling the dependencies essentially *is* that mechanism. If you still have the original wheelhouse cache around, it shouldn't be substantially slower than a plain filesystem copy (since it will just be a matter of unpacking the archived wheel files again, rather than doing any binary builds). For a more literal copy operation, there's "pew cp", but the documentation for that has "this may result in a subtly broken virtual environment" caveats on it for good reasons :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Hi all, I'm using PyCharm CE as my development environment, on a macintosh running mac OS 10.12.3. Here's a screenshot of what happens when I run python3 -m site. [image: Inline image 1] As an explanation - here's the exact processes that I'd run to achieve the trouble (and how I fixed it). After installing the current version with a pip3 install in terminal, I went to the python website, and downloaded the 3.6.2 package, and ran the installer. After realizing that my 3.6.1 modules weren't importable by the 3.6.2 version, I attempted to uninstall 3.6.2 using this method: https://stackoverflow.com/questions/3819449/how-to-uninstall-python-2-7-on-a... This removed 3.6.2 files but somehow was still running 3.6.2 when I booted 'python3' from terminal. It was at this point when I reached out to you guys. I then heeded your advice. I reinstalled 3.6.2, and migrated the modules from the 'site-packages' of 3.6.1 to 'site-packages' in 3.6.2. Through this process I found that 3.6.1 had somehow been installed in a different disc location than the entire 3.6 build, which contained 3.6.2. After migrating the packages to the real 3.6 build containing 3.6.2, I left the remaining 3.6.1 location on my disc. Here's what I still see when trying to configure the PyCharm CE interpreter. [image: Inline image 2] Everything seems to be working now though. Thanks again for your help guys - let me know if you have any additional questions and I'll do my best to furnish an answer. I'm studying applications of tensorflow in financial prediction models at the moment. If any of you have read any academic papers on the topic that stand out, I'd be delighted if you might send me a link, or a way I could find it! Additionally, if you happen to know another Python community member who's researching the same topic, it would be great to get in touch. Best regards, Ian On Fri, Aug 18, 2017 at 10:57 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 19 August 2017 at 14:03, Chris Jerdonek <chris.jerdonek@gmail.com> wrote:
I would just like to point out that the normal workflow for pyenv [1], which seems very popular (and which I use), is to do parallel installs rather than upgrades. The OP didn't actually say "upgrade," so it's possible they're using a tool like pyenv.
Right, it was the references to sudo-remove, rebooting, and python3 that indicated to me we were talking system wide operations, rather than any of the environment management tools like pew, pyenv, conda, etc. That's why I'm somewhat puzzled, since even a raw "make altinstall" will overwrite an existing installation of that version.
The most likely candidate for a system level kinda-sorta parallel installation would be via the distinction between /usr/local/* and /usr/*, and if it's a Debian or Ubuntu system, that would result in the actual system version being able to see components in dist-packages that would be invisible to an unpatched upstream build, each version having its own site-packages directory, but both of them still sharing a common user site-packages directory.
Hence my interest in the output of "<path-to-python> -m site" for the affected versions: seeing the default contents of sys.path will likely answer the question of what's going on definitively, rather than having to guess based on incomplete information.
I believe the OP's original
questions would also apply to users of pyenv -- i.e. having a simple way to "copy" installed dependencies from one pyenv install to a new one.
As Wayne noted, generating a requirements.txt file with `pip freeze` and then reinstalling the dependencies essentially *is* that mechanism. If you still have the original wheelhouse cache around, it shouldn't be substantially slower than a plain filesystem copy (since it will just be a matter of unpacking the archived wheel files again, rather than doing any binary builds).
For a more literal copy operation, there's "pew cp", but the documentation for that has "this may result in a subtly broken virtual environment" caveats on it for good reasons :)
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 20 August 2017 at 05:27, Ian Hartley <iahartle@ucsd.edu> wrote:
Hi all,
I'm using PyCharm CE as my development environment, on a macintosh running mac OS 10.12.3.
Ah, that would explain why my intuitions were all wrong - I know how the Windows and *nix installation models work (and I believe the latter also covers homebrew on macOS), but not how the macOS framework installers work :) While pip+wheels should cover most of your needs for financial analytics in Python these days, it may still be worth your while to spend a bit of time poking around at conda if you think you may ever need components written in other languages like R: https://conda.io/docs/ Cheers, Nick. P.S. As far as your research topic goes, while I'm not aware of anyone specifically researching tensorflow in that context, looking through Dr Yves Hilpisch's publication list at http://hilpisch.com/#/11 may point you towards events where that kind of material may have been presented. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 20 Aug 2017, at 05:56, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 20 August 2017 at 05:27, Ian Hartley <iahartle@ucsd.edu <mailto:iahartle@ucsd.edu>> wrote: Hi all,
I'm using PyCharm CE as my development environment, on a macintosh running mac OS 10.12.3.
Ah, that would explain why my intuitions were all wrong - I know how the Windows and *nix installation models work (and I believe the latter also covers homebrew on macOS), but not how the macOS framework installers work :)
FWIW: The framework installers on python.org <http://python.org/> work just like the other installers on upgrades: installing 3.6.2 on a system already running 3.6.1 replaces the 3.6.1 installation and doesn’t perform a side-by-side installation. The major differences w.r.t. a *nix installation are having an “odd” sys.prefix, and having different sys.prefix values for different feature releases. Ronald
Ronald, I was sure that was the case re: python.org installation. It's strange that it installed in two different locations, I really have no idea how that might have come about. FYI I've included a pic of the file pathways when you look at the interpreter photo. I'd first installed python using terminal btw. Ian
On Aug 20, 2017, at 8:55 AM, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 20 Aug 2017, at 05:56, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 20 August 2017 at 05:27, Ian Hartley <iahartle@ucsd.edu> wrote: Hi all,
I'm using PyCharm CE as my development environment, on a macintosh running mac OS 10.12.3.
Ah, that would explain why my intuitions were all wrong - I know how the Windows and *nix installation models work (and I believe the latter also covers homebrew on macOS), but not how the macOS framework installers work :)
FWIW: The framework installers on python.org work just like the other installers on upgrades: installing 3.6.2 on a system already running 3.6.1 replaces the 3.6.1 installation and doesn’t perform a side-by-side installation. The major differences w.r.t. a *nix installation are having an “odd” sys.prefix, and having different sys.prefix values for different feature releases.
Ronald
On 21 August 2017 at 02:16, Ian Hartley <iahartle@ucsd.edu> wrote:
Ronald,
I was sure that was the case re: python.org installation. It's strange that it installed in two different locations, I really have no idea how that might have come about. FYI I've included a pic of the file pathways when you look at the interpreter photo. I'd first installed python using terminal btw.
Investigating a but further, I think that's actually the source of the discrepancy: /usr/local/Cellar is where homebrew puts things, while the python.org installers put things into /Library/Frameworks/. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (5)
-
Chris Jerdonek -
Ian Hartley -
Nick Coghlan -
Ronald Oussoren -
Wayne Werner