Skip to content

Software Containers

Docker and Apptainer/Singularity are container systems that allow users full control over their software environment. You can create your own container image or choose from a set of pre-defined images, and specify that your submitted jobs run within one of these.

For jobs on the PATh Facility, it does not matter whether you provide a Docker or Apptainer/Singularity image. Either is compatible with our system and can be used with little to no modification. Determining factors on when to use Apptainer/Singularity images over Docker images include if an image already exists, external to PATh distribution preferences, and if you have experience building images in one for format and not the other.

Because the PATh Facility is a distributed infrastructure and workloads consists of a large number jobs (and there container executions), it is important to consider how the container image is transferred to the execution nodes. The instructions below contain best practices when it comes to access both Apptainer/Singularity and Docker images.

When using a container for your jobs, the container image is automatically started up when HTCondor matches your job to a slot. The executable provided in the submit script will be run within the context of the container image, having access to software and libraries that were installed to the image, as if they were already on the server where the job is running. Job executables need not (and should not) run any commands to start the container. Nor should the container image contain any entrypoint/cmd - the job is the command to be run in the container.

Exploring Images on the Access Points

Just like it is important to test your codes and jobs at a small scale, you should make sure that your container is working correctly. One way to explore how PATh Facility sees your container images, is to explore them on the PATh Facility access points. Start an interactive session with the Apptainer/Singularity "shell" mode. The recommended command line, similar to how containers are started for jobs, is:

apptainer shell \

This will give you an interactive shell in an Ubuntu 20.04 container, with your current working directory mounted under /srv. You can explore the container and test your code with for example your own inputs from your home directory. Once you are down exploring, exit the container by running exit or with CTRL+D

PATh-Provided Images

The Facilition Team maintains a set of images that are already in a shared Apptainer/Singularity repository. A list of available containers can be found on this page.

If the software you need isn't already supported in a listed container, you can use your own container or any container image in Docker Hub (see sections further below). Once the container you need is in the Apptainer/Singularity repository, your can submit jobs that run within a particular container by listing the container image in the submit file.

+SingularityImage = "/cvmfs/"

<other usual submit file lines>

Custom Apptainer Images

If you already have software in the form of a .sif Apptainer/Singuilarity file, you can stage the .sif file with your job. For small workloads, sending the file with the job is ok:

transfer_input_files = my-custom-image-v1.sif
+SingularityImage = "my-custom-image-v1.sif"

For larger workloads, the image will be resused for each job, and thus the preferred transfer method is an OSDF tool. Store the .sif file under /path-facility/data/$USER/, and then use the OSDF url directly in the +SingularityImage attribute. Note that you can not use shell variable expansion in the submit file - be sure to replace the username with your actual PATh Facility username. Example:

+SingularityImage = "osdf:///path-facility/data/USERNAME/my-custom-image-v1.sif"

<other usual submit file lines>

Be aware that the OSDF aggressively caches the image based on file naming. If you need to do quick changes, please use versioning of the .sif file so that the caches see a "new" name. In this example, replacing my-custom-image-v1.sif with new content will probably mean that some nodes get the old version and some nodes the new version. Prevent this by creating a new file named with v2.

More information on how to create Apptainer/Singularity images can be found in the Apptainer Images Guide.

Custom Docker Images

If you would prefer to create or use an existing Docker Hub container, for example an authoritative container for your software which already exists in Docker Hub, PATh can distribute the image for you via CVMFS. The result is a synchronized copy of the image under /cvmfs/ which is cached and available to the execution nodes. Creating and/or registering a Docker image is described in the Docker Images Guide.

To run a job with a Docker image, use the +SingularityImage to specify the image the job should be using. Example:

+SingularityImage = "/cvmfs/"

<other usual submit file lines>

Another example would be if your Docker Hub username is alice and you created a container called ncbi-blast, and tag v1, added to the shared Singularity repository, your submit file will include:

+SingularityImage = "/cvmfs/"

<other usual submit file lines>

Frequently Asked Questions / Common Issues

FATAL: kernel too old

If you get a FATAL: kernel too old error, it means that the glibc version in the image is too new for the kernel on the host. You can work around this problem by specifying the minimum host kernel. For example, if you want to run the Ubuntu 18.04 image, specfy a minimum host kernel of 3.10.0, formatted as 31000 (major * 10000 + minor * 100 + patch):

Requirements = HAS_SINGULARITY == True && OSG_HOST_KERNEL_VERSION >= 31000