Deploy and Test the App¶
At this point, your app bundle directory structure should look similar to:
fastqc_app/
├── Dockerfile # Done
├── app.json # Done
├── assets
│ ├── lib
│ │ ├── VERSION # Done
│ │ └── container_exec.sh # Do not modify
│ ├── runner.sh # Done
│ └── tester.sh # Do not modify
├── job.json # Done
├── project.ini # Done
├── src
│ └── mycode.py # Optional
└── test
└── SP1.fq # Optional
The final steps are to deploy the app in to the catalog on the tenant, then run a test job to confirm it is working.
Upload Test Data¶
The Tapis app takes FASTQ data as input. Now is a good time to upload some sample data to a storage system.
# Make a directory for test data
$ tapis files mkdir agave://tacc.work.taccuser/ test-data
+--------------+---------------------------------------+
| Field | Value |
+--------------+---------------------------------------+
| name | test-data |
| uuid | 1105399583769563626-242ac112-0001-002 |
| owner | taccuser |
| path | /test-data |
| lastModified | 2020-05-13T18:40:00.027-05:00 |
| source | None |
| status | STAGING_COMPLETED |
| nativeFormat | dir |
| systemId | tacc.work.taccuser |
+--------------+---------------------------------------+
# Upload the SP1.fq file
$ tapis files upload agave://tacc.work.taccuser/test-data/ ./test/SP1.fq
+-------------------+----------+
| Field | Value |
+-------------------+----------+
| uploaded | 1 |
| skipped | 0 |
| messages | 0 |
| bytes_transferred | 21.94 kB |
| elapsed_sec | 3 |
+-------------------+----------+
# Confirm the file exists on the Tapis storage system
$ tapis files list agave://tacc.work.taccuser/test-data/
+--------+--------------+--------+
| name | lastModified | length |
+--------+--------------+--------+
| SP1.fq | 1 minute ago | 22471 |
+--------+--------------+--------+
The test data can be referenced in future calls to the app as:
agave://tacc.work.taccuser/test-data/SP1.fq
Deploy the App¶
The tapis apps deploy command does multiple things. First, it builds and
pushes your docker image to the namespace specified in project.ini. Second,
it uploads the app assets to a Tapis storage space specified by
deploymentSystem and deploymentPath in the app json file. Finally, it
registers the app with the tenant.
Navigate in to the app bundle folder and issue:
$ pwd
~/fastqc_app/
$ ls
Dockerfile app.json assets/ job.json
project.ini src/ test/
$ tapis apps deploy -W ./
+--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| stage | message |
+--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| build | Step 1/4 : FROM tacc/tacc-ubuntu18-mvapich2.3-psm2 |
| build | ---> d554e642ddc5 |
| | |
| build | Step 2/4 : RUN apt-get update && apt-get upgrade -y && apt-get install wget -y && apt-get install zip -y && apt-get install default-jre -y |
| build | ---> Using cache |
| | |
| build | ---> aa1f50856b62 |
| | |
| build | Step 3/4 : RUN wget https://www.bioinformatics.babraham.ac.uk/projects/fastqc/fastqc_v0.11.9.zip && unzip fastqc_v0.11.9.zip && rm fastqc_v0.11.9.zip && chmod +x FastQC/fastqc |
| build | ---> Using cache |
| | |
| build | ---> 3bb6917b68d6 |
| | |
| build | Step 4/4 : ENV PATH "/FastQC/:$PATH" |
| build | ---> Using cache |
| | |
| build | ---> 356927b0a8f6 |
| | |
| build | Successfully built 356927b0a8f6 |
| | |
| build | Successfully tagged taccuser/fastqc_app:0.11.9 |
| | |
| push | The push refers to repository [docker.io/taccuser/fastqc_app] |
| push | 0.11.9: digest: sha256:29eb2fdb1503fdd38ae311dabfc13958f0910253580614dba0d3ac2dd0753e41 size: 4085 |
| upload | assets/runner.sh |
| upload | assets/tester.sh |
| upload | assets/lib/VERSION |
| upload | assets/lib/container_exec.sh |
| create | Created Tapis app fastqc_app-0.11.9 revision 1 |
+--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
If all goes well, you should see a successful message at the end of the log above, and you should see the new app listed in the apps catalog:
$ tapis apps search --name like fastqc
+----------------------------+----------+------------+------------------+----------+-------------------------+
| id | revision | label | shortDescription | isPublic | executionSystem |
+----------------------------+----------+------------+------------------+----------+-------------------------+
| taccuser-fastqc_app-0.11.9 | 2 | fastqc_app | FastQC app | False | tacc.stampede2.taccuser |
+----------------------------+----------+------------+------------------+----------+-------------------------+
Submit a Test Job¶
Submitting a test job has been
described previously
in this how-to guide. Here, testing will be performed in the same way. First,
create an appropriate job.json file.
$ tapis jobs init --no-archive --output fastqc_job.json taccuser-fastqc_app-0.11.9
Which will output the following json, which can be streamed into a file for submission (add the highlighted test data line):
{
"name": "taccuser-fastqc_app-job-1589377205989",
"appId": "taccuser-fastqc_app-0.11.9",
"batchQueue": "skx-normal",
"maxRunTime": "01:00:00",
"memoryPerNode": "1GB",
"nodeCount": 1,
"processorsPerNode": 1,
"archive": false,
"inputs": {
"fastq": "agave://tacc.work.taccuser/test-data/SP1.fq"
},
"parameters": {},
"notifications": [
{
"event": "FINISHED",
"persistent": true,
"url": "taccuser@gmail.com"
},
{
"event": "FAILED",
"persistent": true,
"url": "taccuser@gmail.com"
}
]
}
Then, submit the test job:
$ tapis jobs submit -F fastqc_job.json
+--------+------------------------------------------+
| Field | Value |
+--------+------------------------------------------+
| id | 4e972f77-5bf9-446e-87a2-3541c4ea5745-007 |
| name | taccuser-fastqc_app-job-1589377205989 |
| status | ACCEPTED |
+--------+------------------------------------------+
Finally, when the job status is FINISHED, inspect and retrieve the output:
$ tapis jobs history 4e972f77-5bf9-446e-87a2-3541c4ea5745-007
+-------------------+----------------+------------------------------------------------------------------------------+
| status | created | description |
+-------------------+----------------+------------------------------------------------------------------------------+
| PENDING | 8 minutes ago | Job processing beginning |
| PROCESSING_INPUTS | 8 minutes ago | Identifying input files for staging |
| STAGING_INPUTS | 8 minutes ago | Transferring job input data to execution system |
| STAGING_INPUTS | 8 minutes ago | Job input copy in progress: agave://tacc.work.taccuser/public/SP1.fq to agav |
| | | e://tacc.stampede2.taccuser//scratch/05896/taccuser/taccuser/job-4e972f77-5b |
| | | f9-446e-87a2-3541c4ea5745-007-taccuser-fastqc_app-job-1589377205989/SP1.fq |
| STAGED | 8 minutes ago | Job inputs staged to execution system |
| STAGING_JOB | 8 minutes ago | Staging runtime assets to execution system |
| STAGING_JOB | 8 minutes ago | Fetching application assets from |
| | | agave://tacc.work.taccuser/taccuser/apps/fastqc_app-0.11.9 |
| STAGING_JOB | 8 minutes ago | Staging runtime assets to agave://tacc.stampede2.taccuser//scratch/05896/sd2 |
| | | e0004/taccuser/job-4e972f77-5bf9-446e-87a2-3541c4ea5745-007-taccuser-fastqc_ |
| | | app-job-1589377205989 |
| SUBMITTING | 8 minutes ago | Submitting job to execution system |
| QUEUED | 7 minutes ago | Job queued to execution system queue |
| RUNNING | 3 minutes ago | Job running on execution system |
| CLEANING_UP | 29 seconds ago | Job completed execution |
| FINISHED | 29 seconds ago | Job completed successfully |
+-------------------+----------------+------------------------------------------------------------------------------+
$ tapis jobs outputs list 4e972f77-5bf9-446e-87a2-3541c4ea5745-007
+------------------------------------------------------------------------------------+-----------------+--------+
| name | lastModified | length |
+------------------------------------------------------------------------------------+-----------------+--------+
| SP1.fq | 21 minutes ago | 22471 |
| SP1_fastqc.html | 17 minutes ago | 561767 |
| SP1_fastqc.zip | 17 minutes ago | 420233 |
| container_exec.log | 17 minutes ago | 19232 |
| lib | 17 minutes ago | 4096 |
| runner.sh | 17 minutes ago | 875 |
| taccuser-fastqc_app-job-1589377205989-4e972f77-5bf9-446e-87a2-3541c4ea5745-007.err | 21 minutes ago | 372 |
| taccuser-fastqc_app-job-1589377205989-4e972f77-5bf9-446e-87a2-3541c4ea5745-007.out | 21 minutes ago | 29 |
| taccuser-fastqc_app-job-1589377205989.ipcexe | 21 minutes ago | 2772 |
| tester.sh | 21 minutes ago | 44 |
+------------------------------------------------------------------------------------+-----------------+--------+
$ tapis jobs outputs download 4e972f77-5bf9-446e-87a2-3541c4ea5745-007
Walking remote resource...
Found 13 file(s) in 5s
Downloading .agave.archive...
Downloading .agave.log...
Downloading container_exec.log...
Downloading container_exec.sh...
Downloading VERSION...
Downloading runner.sh...
Downloading taccuser-fastqc_app-job-1589377205989-4e972f77-5bf9-446e-87a2-3541c4ea5745-007.err...
Downloading taccuser-fastqc_app-job-1589377205989-4e972f77-5bf9-446e-87a2-3541c4ea5745-007.out...
Downloading taccuser-fastqc_app-job-1589377205989.ipcexe...
Downloading SP1.fq...
Downloading SP1_fastqc.html...
Downloading SP1_fastqc.zip...
Downloading tester.sh...
Downloaded 13 files in 61s
+-------------+-------+
| Field | Value |
+-------------+-------+
| downloaded | 13 |
| skipped | 0 |
| messages | 0 |
| elapsed_sec | 66 |
+-------------+-------+
If the file SP1_fastq.html exists, then the run was successful.