Singularity
What is Singularity?
Singularity enables users to have full control of their environment. You can see this presentation [1] for an introduction to Singularity.
What can Singularity do for ENEAGRID/CRESCO users?
If user needs a different version of Python, he can use one of those available by scipy* modules (scipy, scipy3, scipy3intel) that can be loaded by the "module" command. For example:
When an HPC cluster is initially deployed the software choice is a compromise between the hardware requirements, the upgrade status and the completeness of the available software.
Then it comes Singularity with its containers: a container can be used to package entire scientific workflows, software and libraries, and even data.
Singularity permits to run safely containers in an HPC environment, interactively or using batch systems.
This means that you don’t have to ask your ENEAGRID admin to install anything for you:
you can download ready-made containers form the web and run them immediately on CRESCO clusters.
For example you can run a Jupyter Notebook, the web interface to phython, or try the new "Julia" programming language, or a 3D modeller as "Blender".
you can also build your specific container, using a your own Linux workstation, as the build requires to have root authority, and run it on CRESCO,
for example to make use of a Deep Learning package as "TensorFlow". Windows and Mac users can run Linux on a virtual machine as VirtualBox or other similar solutions, on their own systems.
All that is illustrated with examples in the following sections:
How to set up Singularity in user environment
How to search, download and run ready-made containers
How to access local file systems from the container
How to submit Singularity jobs with LSF on CRESCO Cluster
How to build your own container
Warning
For full functionality into ENEAGRID, the working directory for launching Singularity must be into GPFS filesystem and NOT in AFS filesystem.
How set up Singularity in user environment
System
Singularity version
Command
CRESCO6
3.5.2
module load singularity-3.5.2
CRESCO6
3.9.4
module load singularity-3.9.4
CRESCO7
3.9.4
module load singularity-3.9.4
How search, download and run ready-made containers
Some existing containers can be searched and download from web.
The search command is useful to search from web a Container Library for users and containers matching the search query.
For example, suppose we want a container equipped with Julia Programming Language (https://julialang.org), then we can search for:
$ singularity search julia
Found 1 users for 'julia'
library://julian
No collections found for 'julia'
Found 6 containers for 'julia'
library://sylabs/examples/julia.sif
Tags: latest
library://sylabs/examples/julia
Tags: latest
library://dtrudg-utsw/demo/julia
Tags: 20190319 latest
library://sebastian_mc/default/julia
Tags: julia1.1.0
library://gipert/default/julia
library://crown421/default/juliabase
Tags: 1.3.1 latest
If we want download one of the results of search, we can use the pull command:
$ singularity pull library://sylabs/examples/julia.sif
INFO: Downloading library image
155.45 MiB / 155.45 MiB [================] 100.00% 870.25 KiB/s 3m2s
Then we can run the container either by run command:
$ singularity run julia.sif_latest.sif
Hello world!
For full tutorial, visit: https://github.com/sylabs/examples/lang/julia this julia script is embeded in the container.
or by shell command to open a terminal and then launch julia:
$ singularity shell julia.sif_latest.sif
Singularity> julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.0.1 (2018-09-29)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia>
or by exec command to directly launch julia:
$ singularity exec julia.sif_latest.sif julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.0.1 (2018-09-29)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia>
How to access local file systems from the container
Generally speaking, in order to bind a host filesystem path with a container path, the command is:
singularity <command> [–-bind $hostpath:$containerpath] <container>
Warning
For full functionality into ENEAGRID the containers must be launched from GPFS.
To access the GPFS file system, it is sufficient during the launch phase to bind the devices belonging to the GPFS cluster.
For example, in the current configuration of the available ENEAGRID GPFS devices the command to be executed is:
$ singularity shell --bind /gporq2:/gporq2 --bind /gporq3:/gporq3 ubuntu.sif
Singularity> df
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 16384 12 16372 1% /
devtmpfs 98778932 0 98778932 0% /dev
tmpfs 98789084 48587428 50201656 50% /dev/shm
/dev/sda4 90133504 67044352 23089152 75% /tmp
AFS 2147483647 0 2147483647 0% /afs/enea.it/por/user/nameuser
vsd_gporq3 811748818944 543857167360 267891651584 67% /gporq3
tmpfs 16384 12 16372 1% /etc/group
vsd_gporq2 825942343680 610361302016 215581041664 74% /gporq2
Singularity>
The current version of Singularity (v3.5.2) automatically inherits the $HOME variable into a container and it allows read and write access to AFS user area.
For example:
$ singularity shell ubuntu.sif
Singularity> echo $HOME
/afs/enea.it/por/user/nameuser
Singularity> cd $HOME
Singularity> echo "Hello World!" > foo.txt
Singularity> cat foo.txt
Hello world!
To completely access the AFS file system, it needs to build a container with some installed packages and bind some paths. In particular it needs obtain Kerberos tickets and AFS tokens from inside a container [2].
For example, create a recipe file as:
BootStrap: yum
OSVersion: 9
MirrorURL: http://repo.almalinux.org/almalinux/9/BaseOS/x86_64/os/
Include: yum
%runscript
echo "This is what happens when you run the container..."
%post
echo "Hello from inside the container"
yum install -y krb5-workstation sssd-krb5
yum -y install pam-devel
yum -y install ncurses-devel
yum -y install perl-devel
yum -y install krb5-devel
yum -y install perl-ExtUtils-Embed
yum -y install rpm-build
yum -y install flex
yum -y install bison
ln -s /usr/bin/klog.krb5 /usr/bin/klog
%environment
export PATH="/usr/local/bin:$PATH"
export LANG="en_US.UTF-8"
Create the Singularity container using the recipe file.
$ singularity build almalinuxAFS.sif almalinuxAFS.def
Warning
This must be done on a workstation where you have root authority. This capability is not permitted on the CRESCO frontends or compute nodes.
Then run the container by binding some files and paths:
singularity shell --bind /etc/krb5.conf:/etc/krb5.conf --bind /etc/krb5.keytab:/etc/krb5.keytab --bind /proc/fs/openafs/afs_ioctl:/proc/fs/openafs/afs_ioctl --bind /usr/vice/etc:/usr/vice/etc --bind /usr/bin/kinit:/usr/bin/kinit --bind /usr/bin/klog:/usr/bin/klog --bind /usr/bin/tokens:/usr/bin/tokens --bind /afs:/afs --bind /usr/lib64/librokenafs.so.2:/usr/lib64/librokenafs.so.2 almalinuxAFS.sif
Singularity> tokens
Tokens held by the Cache Manager:
User's (AFS ID 28044) rxkad tokens for enea.it [Expires Apr 11 12:44]
--End of list--
Singularity> klist
Ticket cache: FILE:/tmp/krb5cc_28044_w46gjF
Default principal: user@ENEA.IT
Valid starting Expires Service principal
03/12/25 11:44:18 04/11/25 12:44:18 krbtgt/ENEA.IT@ENEA.IT
03/12/25 11:44:18 04/11/25 12:44:18 afs/enea.it@ENEA.IT
03/12/25 11:44:18 04/11/25 12:44:18 host/hpc-por-s1.portici.enea.it@ENEA.IT
Singularity>
Submit Singularity jobs
LSF
Submission of Singularity jobs can only be done via the LSF queuing system. Please study the instructions on using LSF in the User Guide [3]. Suppose we have a container myContainer.sif that when we run it prints the message “This is what happens when you run the container...”. Firstly, we can prepare a script as:
launchSingularity.sh
#!/bin/bash
module load Singularity-3.5.2
queue=container_h24
exe="singularity run --bind /gporq2:/gporq2 --bind /gporq3:/gporq3 myContainer.sif"
bsub -q $queue -e %J.err -o %J.out $exe
Then, when we submit, we obtain in LSF output file, the message:
./launchSingularity.sh
Job <665623> is submitted to queue <container_h24>.
<santogiu@cresco6x002 ~/PFS/por/container> cat 665623.out
Sender: LSF System <lsf@cresco6x006>
Subject: Job 665623: <singularity run --bind /gporq1_minni2:/gporq1_minni2 --bind /gporq2:/gporq2 --bind /gporq3:/gporq3 myContainer.sif> in cluster <portici> Done
[. . .]
Your job looked like:
------------------------------------------------------------
# LSBATCH: User input
singularity run --bind /gporq1_minni2:/gporq1_minni2 --bind /gporq2:/gporq2 --bind /gporq3:/gporq3 myContainer.sif
------------------------------------------------------------
Successfully completed.
Resource usage summary:
CPU time : 0.48 sec.
Max Memory : 30 MB
Average Memory : 4.00 MB
Total Requested Memory : -
Delta Memory : -
Max Processes : 1
Max Threads : 1
The output (if any) follows:
**This is what happens when you run the container...**
PS:
Read file <665623.err> for stderr output of this job.
Warning
This job is submitted on the interactive queue container_h24. If you choose a parallel queue, you have take into account its policy for submitting. For example, for cresco6_48h24 parallel queue, you need use a multiple of 48 cores and “span[ptile=48]” option. In other words:
bsub -q cresco6_48h24 -n 48 -R “span[ptile=48]” -e %J.err -o %J.out $exe
If you choose to use a serial queue, e.g. cresco6h144, you can set a numeber of cores between 1 and 48, For example, with 24 cores:
bsub -q cresco6_h144 -n 24 -e %J.err -o %J.out $exe
How to build your own container
This guide is based on tutorial by University of Arizona [4].
Simple Cases: you can use the pull or build commands to download pre-built images from resources like Singularity Hub or Docker.
This example pulls a container from Singularity Hub
$ singularity pull shub://vsoch/hello-world
INFO: Downloading shub image
59.75 MiB / 59.75 MiB [=================================] 100.00%
$ Singularity run hello-world_latest.sif
RaawwWWWWWRRRR!! Avocado!
This example pulls a container from Docker
$ singularity pull docker://godlovedc/lolcow
INFO: Converting OCI blobs to SIF format
INFO: Starting build...
Getting image source signatures
[…]
INFO: Creating SIF file...
INFO: Build complete: lolcow_latest.sif
$ singularity run lolcow_latest.sif
______________________________________
< Stay away from flying saucers today. >
--------------------------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
Singularity Build: a Almalinux with Tensorflow Example
build is the “Swiss army knife” of container creation. You can use it to download and assemble existing containers from external resources like the Container Library and Docker Hub. You can use it to convert containers between the formats supported by Singularity. And you can use it in conjunction with a Singularity definition file to create a container from scratch and customized it to fit your needs. The build command accepts a target as input and produces a container as output. The target defines the method that build uses to create the container. It can be one of the following:
URI beginning with library:// to build from the Container LibraryURI beginning with docker:// to build from Docker HubURI beginning with shub:// to build from Singularity Hubpath to a existing container on your local machinepath to a directory to build from a sandboxpath to a Singularity definition filebuild can produce containers in two different formats that can be specified as follows.- compressed read-only Singularity Image File (SIF) format suitable for production (default)- writable (ch)root directory called a sandbox for interactive development (--sandbox option)
Because build can accept an existing container as a target and create a container in either supported format you can convert existing containers from one format to another.
This is an example of creating a Singularity image to run code that is not supported on HPC. This example uses Tensorflow but any application could be installed in its place. It also uses Almalinux but it could just as easily be Ubuntu.
Install Singularity on linux workstation - https://sylabs.io/docs/
Create the recipe file (which used to be called a definition file) on a workstation with root authority. Let's call it almaTflow.def
BootStrap: yum
OSVersion: 9
MirrorURL: http://repo.almalinux.org/almalinux/9/BaseOS/x86_64/os/
Include: yum
# Setup: commands executed outside the container before the build starts
%setup
echo "Setting up AlmaLinux 9 container"
# Post: commands executed inside the container during the bootstrap process
%post
echo "Configuring AlmaLinux container..."
# Enable Extra Packages for Enterprise Linux (EPEL)
yum -y install 'dnf-command(config-manager)'
yum config-manager --set-enabled crb
yum -y install epel-release
# Install essential packages
yum -y install vim wget python3 python3-pip
# Upgrade pip and install TensorFlow
pip3 install --upgrade pip
pip3 install tensorflow # For CPU
# pip3 install tensorflow-gpu # Uncomment if you want GPU support
# Create mount points for storage
mkdir -p /extra
mkdir -p /xdisk
echo "Installation completed!"
exit 0
# Runscript: commands executed when the container runs
%runscript
echo "Running AlmaLinux container with TensorFlow..."
python3 -c "import tensorflow as tf; print('TensorFlow Version:', tf.__version__)"
# Test: commands executed at the end of the bootstrap process to verify installation
%test
python3 --version
Create the Singularity container using the recipe file.
Warning
This must be done on a workstation where you have root authority. This capability is not permitted on the CRESCO frontends or compute nodes.
$ singularity build almaTflow.sif almaTflow.def
Copy the new image file to your space on HPC. /extra might be a good location as the image might use up your remaining home. There is a line in the definition file that will create the mount for /extra.
5 .Test with a simple command:
$ singularity exec almaTflow.sif python3 --version
Python 3.9.21
Or slightly more complex create a simple python script called hello.py as:
hello.py
#!/usr/bin/python
import sys
print("Hello World: The Python version is %s.%s.%s" % sys.version_info[:3])
and then:
$ singularity exec almaTflow.sif python3 hello.py
Hello World: The Python version is 3.9.21
$ singularity shell almaTflow.sif
Singularity> python3 hello.py
Hello World: The Python version is 3.9.21
Singularity>
Now test TensorFlow on a GPU node with this Linear Regression example TFlow_example.py:
TFlow_example.py
#Linear Regression Example with TensorFlow v2 library
from __future__ import absolute_import, division, print_function
#
import tensorflow as tf
import numpy as np
rng = np.random
#
# Parameters.
learning_rate = 0.01
training_steps = 1000
display_step = 50
#
# Training Data.
X = np.array([3.3,4.4,5.5,6.71,6.93,4.168,9.779,6.182,7.59,2.167,
7.042,10.791,5.313,7.997,5.654,9.27,3.1])
Y = np.array([1.7,2.76,2.09,3.19,1.694,1.573,3.366,2.596,2.53,1.221,
2.827,3.465,1.65,2.904,2.42,2.94,1.3])
n_samples = X.shape[0]
#
# Weight and Bias, initialized randomly.
W = tf.Variable(rng.randn(), name="weight")
b = tf.Variable(rng.randn(), name="bias")
# Linear regression (Wx + b).
def linear_regression(x):
return W * x + b
# Mean square error.
def mean_square(y_pred, y_true):
return tf.reduce_sum(tf.pow(y_pred-y_true, 2)) / (2 * n_samples)
# Stochastic Gradient Descent Optimizer.
optimizer = tf.optimizers.SGD(learning_rate)
#
# Optimization process.
def run_optimization():
# Wrap computation inside a GradientTape for automatic differentiation.
with tf.GradientTape() as g:
pred = linear_regression(X)
loss = mean_square(pred, Y)
# Compute gradients.
gradients = g.gradient(loss, [W, b])
# Update W and b following gradients.
optimizer.apply_gradients(zip(gradients, [W, b]))
#
# Run training for the given number of steps.
for step in range(1, training_steps + 1):
# Run the optimization to update W and b values.
run_optimization()
if step % display_step == 0:
pred = linear_regression(X)
loss = mean_square(pred, Y)
print("step: %i, loss: %f, W: %f, b: %f" % (step, loss, W.numpy(), b.numpy()))
$ singularity exec --nv centosTflow.sif python3 TFlow_example.py
2020-03-24 20:16:04.688690: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /.Singularity.d/libs
2020-03-24 20:16:04.688752: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: UNKNOWN ERROR (303)
2020-03-24 20:16:04.688824: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (cresco6x001.portici.enea.it): /proc/driver/nvidia/version does not exist
2020-03-24 20:16:04.690416: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2020-03-24 20:16:04.716511: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2100000000 Hz
2020-03-24 20:16:04.721587: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3fac5b0 executing computations on platform Host. Devices:
2020-03-24 20:16:04.721634: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): Host, Default Version
step: 50, loss: 0.611608, W: 0.666194, b: -2.140234
step: 100, loss: 0.550456, W: 0.641768, b: -1.967063
step: 150, loss: 0.496299, W: 0.618780, b: -1.804095
step: 200, loss: 0.448335, W: 0.597148, b: -1.650730
step: 250, loss: 0.405857, W: 0.576790, b: -1.506401
step: 300, loss: 0.368237, W: 0.557631, b: -1.370576
step: 350, loss: 0.334920, W: 0.539602, b: -1.242754
step: 400, loss: 0.305414, W: 0.522634, b: -1.122463
step: 450, loss: 0.279282, W: 0.506667, b: -1.009260
step: 500, loss: 0.256139, W: 0.491640, b: -0.902727
step: 550, loss: 0.235642, W: 0.477499, b: -0.802471
step: 600, loss: 0.217490, W: 0.464191, b: -0.708122
step: 650, loss: 0.201414, W: 0.451667, b: -0.619333
step: 700, loss: 0.187177, W: 0.439880, b: -0.535775
step: 750, loss: 0.174568, W: 0.428789, b: -0.457140
step: 800, loss: 0.163401, W: 0.418351, b: -0.383139
step: 850, loss: 0.153511, W: 0.408528, b: -0.313497
step: 900, loss: 0.144752, W: 0.399283, b: -0.247960
step: 950, loss: 0.136995, W: 0.390584, b: -0.186283
step: 1000, loss: 0.130125, W: 0.382397, b: -0.128241
Building your own Container
This section presumes that you have root authority on a Linux/MAC workstation and that you have Singularity installed. You can follow the instructions at the Sylabs web site: https://sylabs.io/docs/ Or Use HPC Container Maker from Nvidia: https://devblogs.nvidia.com/making-containers-easier-with-hpc-container-maker/ This may be easier as it lets you build a recipe without having to know all the syntax. You just include the building blocks that you need, like Cuda and Infiniband, and it will create the Singularity recipe that you will use to build the container. Again, the build is on your workstation.
Additional Information
- Source URL:
- Links