Containerize the Executable¶
If an image of your executable already exists, and was created by a trusted source, consider using that rather than building your own. You may find existing images on hubs such as Docker Hub, BioContainers, or Quay.io.
This tutorial is a quick and dirty summary of how to build your own Docker image as if there is not one available for your executable. This is not meant to replace the full Docker documentation.
We will continue with the example of FastQC from the previous page.
Choose a Source Image¶
Prerequisite: You should have a Docker ID and docker should be installed on your local machine.
The only dependency for FastQC is a reasonably recent Java Runtime Environment. Thus, most modern Linux OS-es should suffice. The TACC Docker hub organization provides a few base images that would work for this. Pull one manually now so it is in your environment:
$ docker pull tacc/tacc-ubuntu18-mvapich2.3-psm2
A list of available base images can be found here.
Also, now is a good time to prepare a src/
directory in the application
bundle. If your code is a python script, for example, this is where you want to
put it. This directory is not necessary, but nice to have if you want to keep
your executable together with the app assets:
$ cd ~/fastqc-app/
$ mkdir src/
Install and Test Interactively¶
The installation process for fastqc
is extremely simple. It is good practice
to test the installation interactively, and record the steps for a Dockerfile:
# Start an interactive docker session
$ docker run --rm -it tacc/tacc-ubuntu18-mvapich2.3-psm2
# Update and install necessary packages
[docker] $ apt-get update
[docker] $ apt-get upgrade -y
[docker] $ apt-get install wget -y
[docker] $ apt-get install zip -y
[docker] $ apt-get install default-jre -y
# Install FastQC
[docker] $ wget https://www.bioinformatics.babraham.ac.uk/projects/fastqc/fastqc_v0.11.9.zip
[docker] $ unzip fastqc_v0.11.9.zip
[docker] $ rm fastqc_v0.11.9.zip
[docker] $ chmod +x /FastQC/fastqc
After a bit of trial and error, the commands above are a reasonably short path to installing the fastqc executable. You can test it from within the docker image to make sure it is working by, for example:
[docker] $ /FastQC/fastqc -h
FastQC - A high throughput sequence QC analysis tool
SYNOPSIS
fastqc seqfile1 seqfile2 .. seqfileN
fastqc [-o output dir] [--(no)extract] [-f fastq|bam|sam]
[-c contaminant file] seqfile1 .. seqfileN
... etc.
Note on Source Code and Mounting Directories¶
In this instance, we could have downloaded the source zip file for FastQC
directly to the src/
directory of our app bundle, then mounted that directory
within the image, e.g.:
$ cd ~/fastqc-app/src/
$ wget https://www.bioinformatics.babraham.ac.uk/projects/fastqc/fastqc_v0.11.9.zip
$ docker run --rm -it -v $PWD:/opt/src tacc/tacc-ubuntu18-mvapich2.3-psm2
... etc.
That route is perfectly reasonable and can be followed here. In fact, if your app is a standalone python script, for example, this is the best method for including it in your Docker image.
However, some packages have very large zip or tar.gz files (100s of MB), and would be cumbersome to keep in this fastqc app bundle folder. It is up to the app developer to find the balance between completeness of source files and responsible disk usage.
Here, we decide to not download the source permanently. Instead, we make a record of where the source came from. For example:
$ cd ~/fastqc-app/src/
$ echo "Source: https://www.bioinformatics.babraham.ac.uk/projects/fastqc/fastqc_v0.11.9.zip" \
>> README.md
Write the Dockerfile¶
Next, translate the steps required to install your software package into a
resonable Dockerfile
. The Dockerfile
should be located at the root
directory, ~/fastqc-app/Dockerfile
:
FROM tacc/tacc-ubuntu18-mvapich2.3-psm2
RUN apt-get update \
&& apt-get upgrade -y \
&& apt-get install wget -y \
&& apt-get install zip -y \
&& apt-get install default-jre -y
RUN wget https://www.bioinformatics.babraham.ac.uk/projects/fastqc/fastqc_v0.11.9.zip \
&& unzip fastqc_v0.11.9.zip \
&& rm fastqc_v0.11.9.zip \
&& chmod +x FastQC/fastqc
ENV PATH "/FastQC/:$PATH"
Build and Test the Image¶
Navigate to the top of the app directory, ~/fastqc-app/
, and the command to
build a new Docker image is:
$ docker build -f Dockerfile --force-rm -t taccuser/fastqc:0.11.9 ./
Once built, test the new image with an example command:
$ docker run --rm fastqc:0.11.9 fastqc -h
- or -
$ docker run --rm fastqc:0.11.9 perl /FastQC/fastqc -h
Note
Calling the complete path to executables is sometimes safer than relying on PATH environment variables
If you see the FastQC help text, the installation likely was successful. At this time, it might be prudent to test with real data as well. Download some test data into a ~/fastqc-app/tests/ directory:
$ cd ~/fastqc-app/
$ mkdir tests/
# Download random sample data or provide your own
# wget https://molb7621.github.io/workshop/_downloads/SP1.fq
Next, run the FastQC pipeline on the example data:
$ docker run -v $PWD:/data fastqc:0.11.9 perl /FastQC/fastqc /data/SP1.fq
If successful, you should find the output files SP1_fastqc.html
and
SP1_fastqc.zip
in the ~/fastqc-app/tests/
directory.
Push Your Image to the Cloud¶
If you are happy with the tests, push your Docker image to a publicly available repository. It can be your own personal repository as long as it is set to public, and not private. To push to your own repository, make sure it was namespaced with your Docker ID. Then:
$ docker push taccuser/fastqc:0.11.9
Assemble Run Commands¶
The final step is to put instructions in runner.sh
on how the app should be
run. In general, these are the same commands we used for testing above.
Most of the lines in the default file should be left alone. See the last three lines of this file for what should be added:
# Allow over-ride
if [ -z "${CONTAINER_IMAGE}" ]
then
version=$(cat ./_util/VERSION)
CONTAINER_IMAGE="index.docker.io/taccuser/fastqc:${version}"
fi
. lib/container_exec.sh
# Write an excution command below that will run a script or binary inside the
# requested container, assuming that the current working directory is
# mounted in the container as its WORKDIR. In place of 'docker run'
# use 'container_exec' which will handle setup of the container on
# a variety of host environments.
#
# Here is a template:
#
# container_exec ${CONTAINER_IMAGE} COMMAND OPTS INPUTS
#
# Here is an example of counting words in local file 'poems.txt',
# outputting to a file 'wc_out.txt'
#
# container_exec ${CONTAINER_IMAGE} wc poems.txt > wc_out.txt
#
# set -x
# set +x
COMMAND="perl /FastQC/fastqc"
PARAMS="${fastq}"
container_exec ${CONTAINER_IMAGE} ${COMMAND} ${PARAMS}
Update the VERSION
File¶
Finally, put the version / docker tag into the file located at
~/fastqc_app/assets/lib/VERSION
:
$ echo "0.11.9" >> ~/fastqc_app/assets/lib/VERSION
$ cat ~/fastqc_app/assets/lib/VERSION
0.11.9