AWS Glue: Easy_Install usecase
Hi everyone, https://setuptools.readthedocs.io/en/latest/deprecated/easy_install.html instructed me that the mailing list would like to know any use cases where Easy_install is felt needed and I believe I have one. and I do not see any posts about this specific use case. AWS Glue is a serverless runtime environment offered by AWS, usually it runs Spark jobs but Jan 22, 2019 they released Python Shells to run quick snippet codes. they have a fairly comprehensive list of preinstalled libraries and allow you to specify wheels and eggs for anything additional stored on an S3 that you would need, with one limitation - no C-extensions, Cython wheels cannot be installed. for my use-case I needed to access some SFTPs but Paramiko depends on cryptography for SSH shenanigans and cryptography (along with its dependencies cffi, PyNaCL, and bcrypt) are CPython. Turns out if you install at runtime there is no issue - it's just with initial setup that has the limitation. of course you then have to install every time that a process runs during paid runtime, rather than startup but better that then not at all. I tried 2 approaches to install at runtime: subprocess.call(['pip' ...]) and setuptools.command.easy_install.main() subprocess.call to pip installs pure python very nicely, but every time it hit a CPython wheel/tarball - CFFI, PyNaCl, bcrypt, cryptography, and paramiko - took 6-7 minutes EACH just to install (files were provided locally, downloaded from the S3 from previous attempt seeing what would happen if you just gave it CPython ) In a separate run setuptools...easy_install processed everything in 2 seconds. Somewhere between pkg_resources.getdistribution('paramiko').activate(), importing them, and setting host keys lead to Main process starting a little less than 2 minutes later. Required use-case? Not technically Fringe use-case? Fairly Appreciated that this is still around? Very much, 2-minute startup runtime is appreciated over upwards of half an hour on every single run.
participants (1)
-
simon.jones@axtria.com