========================== Singularity ========================== What is Singularity? ------------------------- Singularity enables users to have full control of their environment. You can see this `presentation [1] `_ for an introduction to Singularity. What can Singularity do for ENEAGRID/CRESCO users? ------------------------------------------------------------ If user needs a different version of Python, he can use one of those available by scipy* modules (scipy, scipy3, scipy3intel) that can be loaded by the "module" command. For example: When an HPC cluster is initially deployed the software choice is a compromise between the hardware requirements, the upgrade status and the completeness of the available software. Then it comes Singularity with its containers: a container can be used to package entire scientific workflows, software and libraries, and even data. Singularity permits to run safely containers in an HPC environment, interactively or using batch systems. This means that you don’t have to ask your ENEAGRID admin to install anything for you: 1) you can download ready-made containers form the web and run them immediately on CRESCO clusters. For example you can run a Jupyter Notebook, the web interface to phython, or try the new "Julia" programming language, or a 3D modeller as "Blender". 2) you can also build your specific container, using a your own Linux workstation, as the build requires to have root authority, and run it on CRESCO, for example to make use of a Deep Learning package as "TensorFlow". Windows and Mac users can run Linux on a virtual machine as VirtualBox or other similar solutions, on their own systems. All that is illustrated with examples in the following sections: - How to set up Singularity in user environment - How to search, download and run ready-made containers - How to access local file systems from the container - How to submit Singularity jobs with LSF on CRESCO Cluster - How to build your own container .. warning:: For full functionality into ENEAGRID, the working directory for launching Singularity must be into GPFS filesystem and NOT in AFS filesystem. How set up Singularity in user environment ------------------------------------------ =============== =================== ============================= System Singularity version Command --------------- ------------------- ----------------------------- CRESCO6 3.5.2 module load singularity-3.5.2 --------------- ------------------- ----------------------------- CRESCO6 3.9.4 module load singularity-3.9.4 --------------- ------------------- ----------------------------- CRESCO7 3.9.4 module load singularity-3.9.4 =============== =================== ============================= How search, download and run ready-made containers -------------------------------------------------- Some existing containers can be searched and download from web. - The search command is useful to search from web a Container Library for users and containers matching the search query. For example, suppose we want a container equipped with Julia Programming Language (https://julialang.org), then we can search for: .. code-block:: bash $ singularity search julia Found 1 users for 'julia' library://julian No collections found for 'julia' Found 6 containers for 'julia' library://sylabs/examples/julia.sif Tags: latest library://sylabs/examples/julia Tags: latest library://dtrudg-utsw/demo/julia Tags: 20190319 latest library://sebastian_mc/default/julia Tags: julia1.1.0 library://gipert/default/julia library://crown421/default/juliabase Tags: 1.3.1 latest - If we want download one of the results of search, we can use the pull command: .. code-block:: bash $ singularity pull library://sylabs/examples/julia.sif INFO: Downloading library image 155.45 MiB / 155.45 MiB [================] 100.00% 870.25 KiB/s 3m2s - Then we can run the container either by run command: .. code-block:: bash $ singularity run julia.sif_latest.sif Hello world! For full tutorial, visit: https://github.com/sylabs/examples/lang/julia this julia script is embeded in the container. or by shell command to open a terminal and then launch julia: .. code-block:: bash $ singularity shell julia.sif_latest.sif Singularity> julia _ _ _ _(_)_ | Documentation: https://docs.julialang.org (_) | (_) (_) | _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help. | | | | | | |/ _` | | | | |_| | | | (_| | | Version 1.0.1 (2018-09-29) _/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release |__/ | julia> or by exec command to directly launch julia: .. code-block:: bash $ singularity exec julia.sif_latest.sif julia _ _ _ _(_)_ | Documentation: https://docs.julialang.org (_) | (_) (_) | _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help. | | | | | | |/ _` | | | | |_| | | | (_| | | Version 1.0.1 (2018-09-29) _/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release |__/ | julia> How to access local file systems from the container --------------------------------------------------- Generally speaking, in order to bind a host filesystem path with a container path, the command is: .. code-block:: bash singularity [–-bind $hostpath:$containerpath] .. warning:: For full functionality into ENEAGRID the containers must be launched from GPFS. - To access the GPFS file system, it is sufficient during the launch phase to bind the devices belonging to the GPFS cluster. For example, in the current configuration of the available ENEAGRID GPFS devices the command to be executed is: .. code-block:: bash $ singularity shell --bind /gporq2:/gporq2 --bind /gporq3:/gporq3 ubuntu.sif Singularity> df Filesystem 1K-blocks Used Available Use% Mounted on overlay 16384 12 16372 1% / devtmpfs 98778932 0 98778932 0% /dev tmpfs 98789084 48587428 50201656 50% /dev/shm /dev/sda4 90133504 67044352 23089152 75% /tmp AFS 2147483647 0 2147483647 0% /afs/enea.it/por/user/nameuser vsd_gporq3 811748818944 543857167360 267891651584 67% /gporq3 tmpfs 16384 12 16372 1% /etc/group vsd_gporq2 825942343680 610361302016 215581041664 74% /gporq2 Singularity> - The current version of Singularity (v3.5.2) automatically inherits the $HOME variable into a container and it allows read and write access to AFS user area. For example: .. code-block:: bash $ singularity shell ubuntu.sif Singularity> echo $HOME /afs/enea.it/por/user/nameuser Singularity> cd $HOME Singularity> echo "Hello World!" > foo.txt Singularity> cat foo.txt Hello world! To completely access the **AFS** file system, it needs to build a container with some installed packages and bind some paths. In particular it needs obtain **Kerberos tickets and AFS tokens from inside a container** `[2] `_. For example, create a recipe file as: .. code-block:: bash BootStrap: yum OSVersion: 9 MirrorURL: http://repo.almalinux.org/almalinux/9/BaseOS/x86_64/os/ Include: yum %runscript echo "This is what happens when you run the container..." %post echo "Hello from inside the container" yum install -y krb5-workstation sssd-krb5 yum -y install pam-devel yum -y install ncurses-devel yum -y install perl-devel yum -y install krb5-devel yum -y install perl-ExtUtils-Embed yum -y install rpm-build yum -y install flex yum -y install bison ln -s /usr/bin/klog.krb5 /usr/bin/klog %environment export PATH="/usr/local/bin:$PATH" export LANG="en_US.UTF-8" Create the Singularity container using the recipe file. .. code-block:: bash $ singularity build almalinuxAFS.sif almalinuxAFS.def .. warning:: This must be done on a workstation where you have root authority. This capability is not permitted on the CRESCO frontends or compute nodes. Then run the container by binding some files and paths: .. code-block:: singularity shell --bind /etc/krb5.conf:/etc/krb5.conf --bind /etc/krb5.keytab:/etc/krb5.keytab --bind /proc/fs/openafs/afs_ioctl:/proc/fs/openafs/afs_ioctl --bind /usr/vice/etc:/usr/vice/etc --bind /usr/bin/kinit:/usr/bin/kinit --bind /usr/bin/klog:/usr/bin/klog --bind /usr/bin/tokens:/usr/bin/tokens --bind /afs:/afs --bind /usr/lib64/librokenafs.so.2:/usr/lib64/librokenafs.so.2 almalinuxAFS.sif Singularity> tokens Tokens held by the Cache Manager: User's (AFS ID 28044) rxkad tokens for enea.it [Expires Apr 11 12:44] --End of list-- Singularity> klist Ticket cache: FILE:/tmp/krb5cc_28044_w46gjF Default principal: user@ENEA.IT Valid starting Expires Service principal 03/12/25 11:44:18 04/11/25 12:44:18 krbtgt/ENEA.IT@ENEA.IT 03/12/25 11:44:18 04/11/25 12:44:18 afs/enea.it@ENEA.IT 03/12/25 11:44:18 04/11/25 12:44:18 host/hpc-por-s1.portici.enea.it@ENEA.IT Singularity> Submit Singularity jobs ------------------------ LSF ~~~~~~~~ Submission of Singularity jobs can only be done via the LSF queuing system. Please study the instructions on using LSF in the `User Guide [3] `_. Suppose we have a container myContainer.sif that when we run it prints the message “This is what happens when you run the container...”. Firstly, we can prepare a script as: .. code-block:: bash launchSingularity.sh #!/bin/bash module load Singularity-3.5.2 queue=container_h24 exe="singularity run --bind /gporq2:/gporq2 --bind /gporq3:/gporq3 myContainer.sif" bsub -q $queue -e %J.err -o %J.out $exe Then, when we submit, we obtain in LSF output file, the message: .. code-block:: bash ./launchSingularity.sh Job <665623> is submitted to queue . cat 665623.out Sender: LSF System Subject: Job 665623: in cluster Done [. . .] Your job looked like: ------------------------------------------------------------ # LSBATCH: User input singularity run --bind /gporq1_minni2:/gporq1_minni2 --bind /gporq2:/gporq2 --bind /gporq3:/gporq3 myContainer.sif ------------------------------------------------------------ Successfully completed. Resource usage summary: CPU time : 0.48 sec. Max Memory : 30 MB Average Memory : 4.00 MB Total Requested Memory : - Delta Memory : - Max Processes : 1 Max Threads : 1 The output (if any) follows: **This is what happens when you run the container...** PS: Read file <665623.err> for stderr output of this job. .. warning:: This job is submitted on the interactive queue container_h24. If you choose a parallel queue, you have take into account its policy for submitting. For example, for cresco6_48h24 parallel queue, you need use a multiple of 48 cores and “span[ptile=48]” option. In other words: bsub -q cresco6_48h24 -n 48 -R “span[ptile=48]” -e %J.err -o %J.out $exe If you choose to use a serial queue, e.g. cresco6h144, you can set a numeber of cores between 1 and 48, For example, with 24 cores: bsub -q cresco6_h144 -n 24 -e %J.err -o %J.out $exe How to build your own container -------------------------------- This guide is based on tutorial by University of Arizona `[4] `_. - Simple Cases: you can use the pull or build commands to download pre-built images from resources like Singularity Hub or Docker. This example pulls a container from Singularity Hub .. code-block:: bash $ singularity pull shub://vsoch/hello-world INFO: Downloading shub image 59.75 MiB / 59.75 MiB [=================================] 100.00% $ Singularity run hello-world_latest.sif RaawwWWWWWRRRR!! Avocado! This example pulls a container from Docker .. code-block:: bash $ singularity pull docker://godlovedc/lolcow INFO: Converting OCI blobs to SIF format INFO: Starting build... Getting image source signatures […] INFO: Creating SIF file... INFO: Build complete: lolcow_latest.sif $ singularity run lolcow_latest.sif ______________________________________ < Stay away from flying saucers today. > -------------------------------------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || || Singularity Build: a Almalinux with Tensorflow Example ----------------------------------------------------------- build is the “Swiss army knife” of container creation. You can use it to download and assemble existing containers from external resources like the `Container Library `_ and `Docker Hub `_. You can use it to convert containers between the formats supported by Singularity. And you can use it in conjunction with a Singularity definition file to create a container from scratch and customized it to fit your needs. The build command accepts a target as input and produces a container as output. The target defines the method that build uses to create the container. It can be one of the following: | URI beginning with library:// to build from the Container Library | URI beginning with docker:// to build from Docker Hub | URI beginning with shub:// to build from Singularity Hub | path to a existing container on your local machine | path to a directory to build from a sandbox | path to a Singularity definition file | build can produce containers in two different formats that can be specified as follows. | - compressed read-only Singularity Image File (SIF) format suitable for production (default) | - writable (ch)root directory called a sandbox for interactive development (--sandbox option) Because build can accept an existing container as a target and create a container in either supported format you can convert existing containers from one format to another. This is an example of creating a Singularity image to run code that is not supported on HPC. This example uses Tensorflow but any application could be installed in its place. It also uses Almalinux but it could just as easily be Ubuntu. 1. Install Singularity on linux workstation - https://sylabs.io/docs/ 2. Create the recipe file (which used to be called a definition file) on a workstation with root authority. Let's call it almaTflow.def .. code-block:: bash BootStrap: yum OSVersion: 9 MirrorURL: http://repo.almalinux.org/almalinux/9/BaseOS/x86_64/os/ Include: yum # Setup: commands executed outside the container before the build starts %setup echo "Setting up AlmaLinux 9 container" # Post: commands executed inside the container during the bootstrap process %post echo "Configuring AlmaLinux container..." # Enable Extra Packages for Enterprise Linux (EPEL) yum -y install 'dnf-command(config-manager)' yum config-manager --set-enabled crb yum -y install epel-release # Install essential packages yum -y install vim wget python3 python3-pip # Upgrade pip and install TensorFlow pip3 install --upgrade pip pip3 install tensorflow # For CPU # pip3 install tensorflow-gpu # Uncomment if you want GPU support # Create mount points for storage mkdir -p /extra mkdir -p /xdisk echo "Installation completed!" exit 0 # Runscript: commands executed when the container runs %runscript echo "Running AlmaLinux container with TensorFlow..." python3 -c "import tensorflow as tf; print('TensorFlow Version:', tf.__version__)" # Test: commands executed at the end of the bootstrap process to verify installation %test python3 --version 3. Create the Singularity container using the recipe file. .. warning:: This must be done on a workstation where you have root authority. This capability is not permitted on the CRESCO frontends or compute nodes. .. code-block:: bash $ singularity build almaTflow.sif almaTflow.def 4. Copy the new image file to your space on HPC. /extra might be a good location as the image might use up your remaining home. There is a line in the definition file that will create the mount for /extra. 5 .Test with a simple command: .. code-block:: bash $ singularity exec almaTflow.sif python3 --version Python 3.9.21 6. Or slightly more complex create a simple python script called hello.py as: .. code-block:: bash hello.py #!/usr/bin/python import sys print("Hello World: The Python version is %s.%s.%s" % sys.version_info[:3]) and then: .. code-block:: bash $ singularity exec almaTflow.sif python3 hello.py Hello World: The Python version is 3.9.21 $ singularity shell almaTflow.sif Singularity> python3 hello.py Hello World: The Python version is 3.9.21 Singularity> 7. Now test TensorFlow on a GPU node with this Linear Regression example TFlow_example.py: .. code-block:: bash TFlow_example.py #Linear Regression Example with TensorFlow v2 library from __future__ import absolute_import, division, print_function # import tensorflow as tf import numpy as np rng = np.random # # Parameters. learning_rate = 0.01 training_steps = 1000 display_step = 50 # # Training Data. X = np.array([3.3,4.4,5.5,6.71,6.93,4.168,9.779,6.182,7.59,2.167, 7.042,10.791,5.313,7.997,5.654,9.27,3.1]) Y = np.array([1.7,2.76,2.09,3.19,1.694,1.573,3.366,2.596,2.53,1.221, 2.827,3.465,1.65,2.904,2.42,2.94,1.3]) n_samples = X.shape[0] # # Weight and Bias, initialized randomly. W = tf.Variable(rng.randn(), name="weight") b = tf.Variable(rng.randn(), name="bias") # Linear regression (Wx + b). def linear_regression(x): return W * x + b # Mean square error. def mean_square(y_pred, y_true): return tf.reduce_sum(tf.pow(y_pred-y_true, 2)) / (2 * n_samples) # Stochastic Gradient Descent Optimizer. optimizer = tf.optimizers.SGD(learning_rate) # # Optimization process. def run_optimization(): # Wrap computation inside a GradientTape for automatic differentiation. with tf.GradientTape() as g: pred = linear_regression(X) loss = mean_square(pred, Y) # Compute gradients. gradients = g.gradient(loss, [W, b]) # Update W and b following gradients. optimizer.apply_gradients(zip(gradients, [W, b])) # # Run training for the given number of steps. for step in range(1, training_steps + 1): # Run the optimization to update W and b values. run_optimization() if step % display_step == 0: pred = linear_regression(X) loss = mean_square(pred, Y) print("step: %i, loss: %f, W: %f, b: %f" % (step, loss, W.numpy(), b.numpy())) .. code-block:: bash $ singularity exec --nv centosTflow.sif python3 TFlow_example.py 2020-03-24 20:16:04.688690: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /.Singularity.d/libs 2020-03-24 20:16:04.688752: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: UNKNOWN ERROR (303) 2020-03-24 20:16:04.688824: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (cresco6x001.portici.enea.it): /proc/driver/nvidia/version does not exist 2020-03-24 20:16:04.690416: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA 2020-03-24 20:16:04.716511: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2100000000 Hz 2020-03-24 20:16:04.721587: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3fac5b0 executing computations on platform Host. Devices: 2020-03-24 20:16:04.721634: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): Host, Default Version step: 50, loss: 0.611608, W: 0.666194, b: -2.140234 step: 100, loss: 0.550456, W: 0.641768, b: -1.967063 step: 150, loss: 0.496299, W: 0.618780, b: -1.804095 step: 200, loss: 0.448335, W: 0.597148, b: -1.650730 step: 250, loss: 0.405857, W: 0.576790, b: -1.506401 step: 300, loss: 0.368237, W: 0.557631, b: -1.370576 step: 350, loss: 0.334920, W: 0.539602, b: -1.242754 step: 400, loss: 0.305414, W: 0.522634, b: -1.122463 step: 450, loss: 0.279282, W: 0.506667, b: -1.009260 step: 500, loss: 0.256139, W: 0.491640, b: -0.902727 step: 550, loss: 0.235642, W: 0.477499, b: -0.802471 step: 600, loss: 0.217490, W: 0.464191, b: -0.708122 step: 650, loss: 0.201414, W: 0.451667, b: -0.619333 step: 700, loss: 0.187177, W: 0.439880, b: -0.535775 step: 750, loss: 0.174568, W: 0.428789, b: -0.457140 step: 800, loss: 0.163401, W: 0.418351, b: -0.383139 step: 850, loss: 0.153511, W: 0.408528, b: -0.313497 step: 900, loss: 0.144752, W: 0.399283, b: -0.247960 step: 950, loss: 0.136995, W: 0.390584, b: -0.186283 step: 1000, loss: 0.130125, W: 0.382397, b: -0.128241 Building your own Container --------------------------- This section presumes that you have root authority on a Linux/MAC workstation and that you have Singularity installed. You can follow the instructions at the Sylabs web site: https://sylabs.io/docs/ Or Use HPC Container Maker from Nvidia: https://devblogs.nvidia.com/making-containers-easier-with-hpc-container-maker/ This may be easier as it lets you build a recipe without having to know all the syntax. You just include the building blocks that you need, like Cuda and Infiniband, and it will create the Singularity recipe that you will use to build the container. Again, the build is on your workstation. Additional Information ---------------------- Source URL: | Links | https://sylabs.io | https://www.afs.enea.it/software/singularity/introduction-to-singularity.pdf | https://confluence.desy.de/display/IS/Using+Kerberos+Tickets+and+AFS+Tokens+from+Inside+a+Container | https://www.afs.enea.it/project/eneagrid/Resources_en/CRESCO_documents/index.html | https://public.confluence.arizona.edu/display/UAHPC/Singularity+Tutorials