========================== Apptainer ========================== What is Apptainer? ------------------------- Apptainer is the evolution of Singularity and allows you to create customized containers, providing greater control over the execution environment. What can Apptainer do for ENEAGRID/CRESCO users? ------------------------------------------------------------ - Run pre-built containers: download images from the web and run them on CRESCO7 (e.g., Jupyter Notebook, Julia). - Create a custom container: set up your environment on Linux and use it on HPC (e.g., TensorFlow). - Isolate your software environment: avoid version conflicts and ensure reproducibility. - Work securely: run containers on HPC clusters in interactive or batch mode. How set up Apptainer in user environment ------------------------------------------- =============== =================== ============================= System Apptainer version Command --------------- ------------------- ----------------------------- CRESCO7 1.3.6 module load apptainer-1.3.6 =============== =================== ============================= How to search, download, and run ready-made containers --------------------------------------------------------- Some existing containers can be searched and downloaded from the web. The **search** command is useful for finding containers and users in a Container Library that match the search query. For example, if we need a container with the Julia programming language (https://julialang.org), we can search for it using: .. code-block:: bash apptainer search julia Found 18 container images for amd64 matching "julia": library://cylon-x/lcsb-biocore/julia-base:latest library://dtrudg-utsw/demo/julia:20190319 demo for 2019-03-20 library://dtrudg-utsw/demo/julia:latest library://francesco.esposito/fraespo/julia:latest library://iaopan/iaopan/julia-latest:latest library://juliahoeglund/collection/maftools:latest library://julian_spaeth/default/partea:latest library://julianh/helmholtz-benchmark/graphprot2:latest library://laurentpagnier/default/julia:latest library://millironx/default/julia:1,1.6,1.6.2,latest library://millironx/default/julia:sha256.15d7efa05c62b0d5917af52db06f8f29d5ab96298d33d636ebe10de027f83aab Nextflow-compatible image for Julia v1.6.1. See https://github.com/JuliaLang/julia/blob/master/HISTORY.md#julia-v16-release-notes for changes. library://sebastian_mc/default/julia:julia1.1.0 library://soymintc/julia/mmctm-jl_1.6.5:latest library://soymintc/julia/tmmm-julia.sif:latest library://soymintc/sig/julia:1.7.2 library://sylabs/examples/julia.sif:latest library://sylabs/examples/julia:latest library://valentinwust/general/julia:latest If we want download one of the results of search, we can use the pull command: .. code-block:: bash $ apptainer pull library://cylon-x/lcsb-biocore/julia-base:latest INFO: Downloading library image 65.0MiB / 150.1MiB [========>----------------] 43 % 8.7 MiB/s 9s Then we can run the container either by run command: .. code-block:: bash $ apptainer run --unsquash julia-base_latest.sif INFO: squashfuse not found, will not be able to mount SIF or other squashfs files INFO: gocryptfs not found, will not be able to use gocryptfs INFO: Converting SIF file to temporary sandbox... _ _ _ _(_)_ | Documentation: https://docs.julialang.org (_) | (_) (_) | _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help. | | | | | | |/ _` | | | | |_| | | | (_| | | Version 1.6.1 (2021-04-23) _/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release |__/ | julia> or by shell command (or exec) to open a terminal and then launch julia: .. code-block:: bash $ apptainer shell --unsquash julia-base_latest.sif INFO: squashfuse not found, will not be able to mount SIF or other squashfs files INFO: gocryptfs not found, will not be able to use gocryptfs INFO: Converting SIF file to temporary sandbox... Apptainer> julia _ _ _ _(_)_ | Documentation: https://docs.julialang.org (_) | (_) (_) | _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help. | | | | | | |/ _` | | | | |_| | | | (_| | | Version 1.6.1 (2021-04-23) _/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release |__/ | julia> exit() ------------------------------------------------------------------------------------- $ apptainer exec --unsquash julia-base_latest.sif julia INFO: squashfuse not found, will not be able to mount SIF or other squashfs files INFO: gocryptfs not found, will not be able to use gocryptfs INFO: Converting SIF file to temporary sandbox... _ _ _ _(_)_ | Documentation: https://docs.julialang.org (_) | (_) (_) | _ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help. | | | | | | |/ _` | | | | |_| | | | (_| | | Version 1.6.1 (2021-04-23) _/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release |__/ | julia> How to access local file systems --------------------------------- To bind a filesystem from the host to a path in the container, the command syntax is: .. code-block:: bash apptainer shell --unsquash --bind $hostpath:$containerpath Here is an example with the LUSTRE filesystem: .. code-block:: bash apptainer shell --unsquash --bind /var/tmp/mtab:/etc/mtab --bind /lustpor1:/lustpor1/ julia-base_latest.sif .. note:: The partitions will be identified as simple folders; therefore, they will not be visible with the "df -h" command but will be available to the user. You can also do it with GPFS. .. code-block:: bash apptainer shell --unsquash --bind /var/tmp/mtab:/etc/mtab --bind /gporq2:/gporq2 --bind /gporq3:/gporq3 julia-base_latest.sif .. warning:: The filesystem is only available on the FRONTENDs "cresco7x001" and "cresco7x002". The current version of Apptainer automatically inherits the $HOME variable into a container and it allows read and write access to AFS user area: .. code-block:: bash apptainer shell --unsquash julia_latest.sif Apptainer> echo $HOME /afs/enea.it/por/user/USER .. note:: If you are using a specific container that does not inherit the $HOME variable, you can import it with this command: apptainer shell --unsquash --bind $HOME:/user To completely access the **AFS** file system, it needs to build a container with some installed packages and bind some paths. In particular it needs obtain **Kerberos tickets and AFS tokens from inside a container** `[2] `_. For example, create a recipe file as: .. code-block:: bash BootStrap: yum OSVersion: 9 MirrorURL: http://repo.almalinux.org/almalinux/9/BaseOS/x86_64/os/ Include: yum %runscript echo "This is what happens when you run the container..." %post echo "Hello from inside the container" yum install -y krb5-workstation sssd-krb5 yum -y install pam-devel yum -y install ncurses-devel yum -y install perl-devel yum -y install krb5-devel yum -y install perl-ExtUtils-Embed yum -y install rpm-build yum -y install flex yum -y install bison ln -s /usr/bin/klog.krb5 /usr/bin/klog %environment export PATH="/usr/local/bin:$PATH" export LANG="en_US.UTF-8" Create the Apptainer container using the recipe file. .. code-block:: bash apptainer build almalinuxAFS.sif almalinuxAFS.def .. note:: This must be done on a workstation where you have root authority. This capability is not permitted on the CRESCO frontends or compute nodes. Then run the container by binding some files and paths: .. code-block:: bash apptainer shell --unsquash --bind /etc/krb5.conf:/etc/krb5.conf --bind /etc/krb5.keytab:/etc/krb5.keytab --bind /proc/fs/openafs/afs_ioctl:/proc/fs/openafs/afs_ioctl --bind /usr/vice/etc:/usr/vice/etc --bind /usr/bin/kinit:/usr/bin/kinit --bind /usr/bin/klog:/usr/bin/klog --bind /usr/bin/tokens:/usr/bin/tokens --bind /afs:/afs --bind /usr/lib64/librokenafs.so.2:/usr/lib64/librokenafs.so.2 almalinuxAFS.sif Apptainer> tokens Tokens held by the Cache Manager: User's (AFS ID 28044) rxkad tokens for enea.it [Expires Apr 12 11:29] --End of list-- Apptainer> klist Ticket cache: FILE:/tmp/krb5cc_28044_eE0nJJ Default principal: user@ENEA.IT Valid starting Expires Service principal 03/13/25 10:29:31 04/12/25 11:29:31 krbtgt/ENEA.IT@ENEA.IT 03/13/25 10:29:31 04/12/25 11:29:31 afs/enea.it@ENEA.IT 03/13/25 10:29:32 04/12/25 11:29:31 host/hpc-por-s1.portici.enea.it@ENEA.IT Submit Apptainer jobs ------------------------ SLURM ~~~~~~ To submit Apptainer jobs via SLURM, you need to create a job submission script. Below is a script that runs a Apptainer container (almalinuxAFS.sif) and prints "This is what happens when you run the container...". .. code-block:: bash #!/bin/bash #SBATCH --job-name=container_job #SBATCH --output=%j.out #SBATCH --error=%j.err #SBATCH --partition=cresco7 #SBATCH --ntasks=1 #SBATCH --cpus-per-task=4 #SBATCH --mem=8G #SBATCH --time=24:00:00 module load apptainer-1.3.6 exe="apptainer run --unsquash almalinuxAFS.sif" $exe Submit the job with SLURM .. code-block:: bash sbatch submit_apptainer_slurm.sh Submitted batch job 196346 Check the job status .. code-block:: bash squeue -u $USER How to build your own container -------------------------------- This guide is based on tutorial by University of Arizona `[4] `_. - Simple Cases: you can use the pull or build commands to download pre-built images from resources like Apptainer Hub or Docker. This example pulls a container from Apptainer Hub .. code-block:: bash apptainer pull shub://vsoch/hello-world INFO: Downloading shub image 59.8MiB / 59.8MiB [=================================] 100 % 19.8 MiB/s 0s apptainer run --unsquash hello-world_latest.sif RaawwWWWWWRRRR!! Avocado! This example pulls a container from Docker .. code-block:: bash apptainer pull docker://godlovedc/lolcow INFO: Using cached SIF image --------------------------------------------- apptainer run --unsquash lolcow_latest.sif INFO: gocryptfs not found, will not be able to use gocryptfs INFO: Converting SIF file to temporary sandbox... _________________________________________ / Your depth of comprehension may tend to \ \ make you lax in worldly ways. / ----------------------------------------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || || Apptainer Build: a Almalinux with Tensorflow Example --------------------------------------------------------- build is the “Swiss army knife” of container creation. You can use it to download and assemble existing containers from external resources like the `Container Library `_ and `Docker Hub `_. You can use it to convert containers between the formats supported by Apptainer. And you can use it in conjunction with a Apptainer definition file to create a container from scratch and customized it to fit your needs. The build command accepts a target as input and produces a container as output. The target defines the method that build uses to create the container. It can be one of the following: | URI beginning with library:// to build from the Container Library | URI beginning with docker:// to build from Docker Hub | URI beginning with shub:// to build from Apptainer Hub | path to a existing container on your local machine | path to a directory to build from a sandbox | path to a Apptainer definition file | build can produce containers in two different formats that can be specified as follows. | - compressed read-only Apptainer Image File (SIF) format suitable for production (default) | - writable (ch)root directory called a sandbox for interactive development (--sandbox option) Because build can accept an existing container as a target and create a container in either supported format you can convert existing containers from one format to another. This is an example of creating a Apptainer image to run code that is not supported on HPC. This example uses Tensorflow but any application could be installed in its place. It also uses Almalinux but it could just as easily be Ubuntu. 1. Install Apptainer on linux workstation - https://sylabs.io/docs/ 2. Create the recipe file (which used to be called a definition file) on a workstation with root authority. Let's call it almaTflow.def .. code-block:: bash BootStrap: yum OSVersion: 9 MirrorURL: http://repo.almalinux.org/almalinux/9/BaseOS/x86_64/os/ Include: yum # Setup: commands executed outside the container before the build starts %setup echo "Setting up AlmaLinux 9 container" # Post: commands executed inside the container during the bootstrap process %post echo "Configuring AlmaLinux container..." # Enable Extra Packages for Enterprise Linux (EPEL) yum -y install 'dnf-command(config-manager)' yum config-manager --set-enabled crb yum -y install epel-release # Install essential packages yum -y install vim wget python3 python3-pip # Upgrade pip and install TensorFlow pip3 install --upgrade pip pip3 install tensorflow # For CPU # pip3 install tensorflow-gpu # Uncomment if you want GPU support # Create mount points for storage mkdir -p /extra mkdir -p /xdisk echo "Installation completed!" exit 0 # Runscript: commands executed when the container runs %runscript echo "Running AlmaLinux container with TensorFlow..." python3 -c "import tensorflow as tf; print('TensorFlow Version:', tf.__version__)" # Test: commands executed at the end of the bootstrap process to verify installation %test python3 --version 3. Create the Apptainer container using the recipe file. .. note:: This must be done on a workstation where you have root authority. This capability is not permitted on the CRESCO frontends or compute nodes. .. code-block:: bash apptainer build almaTflow.sif almaTflow.def 4. Copy the new image file to your space on HPC. /extra might be a good location as the image might use up your remaining home. There is a line in the definition file that will create the mount for /extra. 5. Test with a simple command: .. code-block:: bash apptainer exec almaTflow.sif python3 --version Python 3.9.21 6. Or slightly more complex create a simple python script called hello.py as: .. code-block:: bash #!/usr/bin/python import sys print("Hello World: The Python version is %s.%s.%s" % sys.version_info[:3]) and then: .. code-block:: bash apptainer exec almaTflow.sif python3 hello.py Hello World: The Python version is 3.9.21 Building your own Container --------------------------- This section presumes that you have root authority on a Linux/MAC workstation and that you have Apptainer installed. You can follow the instructions at the Sylabs web site: https://sylabs.io/docs/ Or Use HPC Container Maker from Nvidia: https://devblogs.nvidia.com/making-containers-easier-with-hpc-container-maker/ This may be easier as it lets you build a recipe without having to know all the syntax. You just include the building blocks that you need, like Cuda and Infiniband, and it will create the Apptainer recipe that you will use to build the container. Again, the build is on your workstation. Additional Information ---------------------- Source URL: | Links | https://apptainer.org/ | https://confluence.desy.de/display/IS/Using+Kerberos+Tickets+and+AFS+Tokens+from+Inside+a+Container | https://www.afs.enea.it/project/eneagrid/Resources_en/CRESCO_documents/index.html | https://public.confluence.arizona.edu/display/UAHPC/Singularity+Tutorials