Python

What is Python?

Python is a programming language that lets you work quickly and integrate systems more effectively.

What Python version/environment can users find/set up on ENEAGRID/CRESCO?

On HPC clusters is already available the default system Python:

    Version 2.6.6 on CRESCO4
    <user@cresco4x001 ~> python -V
    Python 2.6.6
    Version 2.7.5 on CRESCO6
    <user@cresco6x001 ~> python -V
    Python 2.7.5

In the following sections is explained:
  • How to set up a different Python version in user environment
  • How to create a local Python environment by virtualenv tool

How to set up a different Python version in user environment

If user needs a different version of Python, he can use one of those available by scipy* modules (scipy, scipy3, scipy3intel) that can be loaded by the "module" command.
For example:

    <user@cresco6x001 ~> python -V
    Python 2.7.5
    <user@cresco6x001 ~> module load scipy
    <user@cresco6x001 ~> python -V
    Python 3.7.0
    <user@cresco6x001 ~> module load scipy3
    <user@cresco6x001 ~> python -V
    Python 3.6.7
    <user@cresco6x001 ~> module load scipy3intel
    <user@cresco6x001 ~> python -V

To summarize:

    System Python version Command
    CRESCO6/CRESCO4 3.7.0 module load scipy
    CRESCO6/CRESCO4 3.6.7 module load scipy3
    CRESCO6/CRESCO4 3.6.9 module load scipy3intel

How to create a local Python environment by virtualenv tool

If user needs to create a local environment, where he can install all Python modules required for a particular software, than the virtualenv (already installed on CRESCO) tool can be a useful, after loading the scipy3 module. By virtualenv user can generate and configure a local Python environment (also specifying the destination directory, such a subfolder of user home) in which he can install all the necessary modules/packages.

On Web user can finds many tutorials, such as 2 and 3.

N.B.: user does not have to install virtualenv as described in the tutorials as it has already been done. He has only load scipy3 module.

For example:

  • Create a subfolder in user home, move on it and load scipy3 module:
    <user@cresco6x001 ~> mkdir myLocalPython
    <user@cresco6x001 ~> cd myLocalPython/
    <user@cresco6x001 ~/myLocalPython>
    <user@cresco6x001 ~> module load scipy3
  • Generate a local (virtual) environment myenv by virtualenv:
    <user@cresco6x001 ~/myLocalPython> virtualenv myenv
    created virtual environment CPython3.6.7.final.0-64 in 2280ms
    creator CPython3Posix(dest=/afs/enea.it/por/user/user/myLocalPython/myenv, clear=False, global=False)
    seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/afs/enea.it/por/user/user/.local/share/virtualenv)
    added seed packages: pip==20.2.2, setuptools==50.2.0, wheel==0.35.1
    activators
    BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
    <user@cresco6x001 ~/myLocalPython> ls
    myenv
  • the local environment:
    <user@cresco6x001 ~/myLocalPython> source myenv/bin/activate
    (myenv) <user@cresco6x001 ~/myLocalPython>
    The myenv in parentheses on the terminal, before the username and hostname, indicates that the myenv environment is active and running.

  • Then use local python and pip to install packages: for example consider to install pandas package in a requirement.txt file.
    (myenv) <user@cresco6x001 ~/myLocalPython> which python
    ~/myLocalPython/myenv/bin/python
    (myenv) <user@cresco6x001 ~/myLocalPython> which pip
    ~/myLocalPython/myenv/bin/pip
    (myenv) <user@cresco6x001 ~/myLocalPython> echo "pandas" > requirements.txt
    (myenv) <user@cresco6x001 ~/myLocalPython> ls
    myenv requirements.txt
    (myenv) <user@cresco6x001 ~/myLocalPython> cat requirements.txt
    pandas
    (myenv) <user@cresco6x001 ~/myLocalPython> pip install -r requirements.txt
    Collecting pandas

    Downloading pandas-1.1.3-cp36-cp36m-manylinux1_x86_64.whl (9.5 MB)

    |████████████████████████████████| 9.5 MB 11.4 MB/s
    Collecting numpy>=1.15.4

    Downloading numpy-1.19.2-cp36-cp36m-manylinux2010_x86_64.whl (14.5 MB)

    |████████████████████████████████| 14.5 MB 29.1 MB/s
    Collecting python-dateutil>=2.7.3
    Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
    Collecting pytz>=2017.2

    Downloading pytz-2020.1-py2.py3-none-any.whl (510 kB)

    |████████████████████████████████| 510 kB 45.0 MB/s
    Collecting six>=1.5
    Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
    Installing collected packages: numpy, six, python-dateutil, pytz, pandas
    Successfully installed numpy-1.19.2 pandas-1.1.3 python-dateutil-2.8.1 pytz-2020.1 six-1.15.0
  • To deactivate virtualenv use deactivate command:
    (myenv) <user@cresco6x001 ~/myLocalPython> deactivate
    <user@cresco6x001 ~/myLocalPython>
    The myenv in parentheses on the terminal, before the username and hostname, disappears.

Additional Information

Source URL:
Links