Create a Private System¶
Many tenants automatically provide their users with private storage and execution systems connecting to TACC resources, but some do not. This guide walks through the process of creating your own private systems. This can also be used to help you connect to non-TACC lab servers, cloud VMs, and clusters.
Gather Relevant Information¶
To register any system in Tapis, you need the hostname and login credentials. The preferred login credentials are username and SSH key pairs. Storage systems (for storing files) require a default path where you have write access. Execution systems (for running jobs) also require a default path where job runtime files will be staged and the job will be executed. If it is an ‘HPC’ type execution system, then you also need information about the queueing system (queue names, limits, etc.).
For this demonstration, we will set up a storage system to access the TACC work directory (via Stampede2):
hostname: stampede2.tacc.utexas.edu
username: taccuser
credentials: <ssh keys>
storage path: /work/01234/taccuser
And we will set up an execution system for the Stampede2 HPC cluster:
hostname: stampede2.tacc.utexas.edu
username: taccuser
credentials: <ssh keys preferred>
storage path: /work/01234/taccuser
job runtime path: /scratch/01234/taccuser
queue_type: SLURM
queue: normal (limits in Stampede2 user guide)
Note
This guide assumes you have the appropriate permissions and credentials to access Stampede2. These can be attained by having an active Stampede2 Allocation
Register a Storage System¶
To register a system, you need to assemble a json description of the above
requirements and some additional metadata. Start by saving the following storage
system template in a file called tacc.work.taccuser.json:
{
"id": "tacc.work.taccuser",
"name": "Storage system for the TACC WORK directory",
"description": "Storage system for the TACC WORK directory via Stampede2",
"type": "STORAGE",
"storage": {
"host": "stampede2.tacc.utexas.edu",
"port": 22,
"protocol": "SFTP",
"rootDir": "/work/01234/taccuser",
"homeDir": "/",
"auth":{
"username":"taccuser",
"publicKey": " <enter public key here> ",
"privateKey": " <enter private key here> ",
"type": "SSHKEYS"
}
}
}
Most fields are fairly self explanatory, but here is a brief breakdown of the important options:
id: a unique identifier and how the system name appears fortapis systems listcommandsname: common display name for the systemdescription: optional long plain text description of the systemtype: can beSTORAGEorEXECUTIONhost: IP address or hostname of systemrootDir: path of the virtual root directory on the remote systemhomeDir: path relative torootDirfortapis files -operationsusername: this is your username for the target systempublicKey: cut and paste your public key hereprivateKey: cut and paste your private key here
Edit the username and paths in the above template to match your username and
work folder. A copy of your public key should be added to the
~/.ssh/authorized_keys file on the remote host. The public and private
key should be pasted on one line each similar to the following. Replace the
line breaks in the private key with the newline character \n:
{
"auth":{
"username": "taccuser",
"publicKey": "ssh-rsa AAAAB3NzaC1yc2EBBAADAQABMQRgQqSuJdTi+VwMif8qouSSEWVduKZHpzOnS1zlknAyYXmQQFcaJ+vNAQayVMTqv+A+1lzxppTdgZ0Dn42EOYWRa6B/IEMPzDuKb7F0qNFiH9m+OZJDYdIWS1rlN1oK32jHUi0xV8kM3KOLf2TIjDBUyZRpMGyQ= user@email.com",
"privateKey": "-----BEGIN RSA PRIVATE KEY-----\nMIIEpAIBAAKCAQEA1Jhi5BNiogg3NtALJepyTz5xS3j/dpYBGf5ERBH0C\n4SCb9VAxOCyb4l+QDrOQnLRX2RV4JjHlw7r8qmc6IvPmk83oTYqYN2NuzMjxI\nsqjVfmJgnF4sPuQy+Pioie9UeekAJRfaJLChZxLfyfppUVNTOOg6rVkERV/n9IDr\nTY2r/B16XtzcjGYvhW35Avy2FlTHvJldwaxmY4UuNey7r9LXAved4nqTj7d\n5PVKgWB8Bu6h5U1EGgnPhFFi8MJCO4/bByqAYdEffC9Y+cWBFq749XNPafid\nDlKFza44RR5Fg86OZxJW7NGoMnIjVYRIcUQIDAQABAoIBAQCovogIBscMW6R/\nfTwM/h3OlUu9EdlVOfygwkq5GfdbPBco291UOmDwN08aryTR8JtVLPO5ZtevX\nTVXVpWtejdLr5aL/R/uYxhxaIoeI7ppQBk3daSNsZia2lRp1j4qil\nyKfy5WxHdzjAhg3gamYtTk981qJIOSR0kQxxz3ax23BN5C/r1uqHK1hFUlCgx\nRrjt2M2/TvFtGZdRmxH4Kdco7IeOtj0xAYS/hGBV4CRa+4zWb3ikNOVxcFL61\nuT/60043DsVI22B5zv3XODtfSjquqlYl5eHZdf+HdL6u91CKrjmvpg80OfQ\ngmPwhOjdAoGBAOtRhXta/Y8X1U+XykaXfVFsfzFsslMtI73XII+nKYdtDFlSl\nLYg6PB5Gk9Q++RdinHzL7DwAXOVWW2nwfoEKxjsYCw4ihYVO/UEG\nqqCeu0X/r9N6u78HfeZEX5XH3+QtR/d9bP2mLjhY8LTAoGBAOdH\nnHnrMpeiEzou+5UC3lKRUN84LX/o9kp6t6WSF5oT7tQEyJKVICgLBOMVbASvXZxncYziYJKIzrqDkC+QXdZpF0x/u04vryDz9ySl9rhBYaD74e+FFXkDImMAQ\nCL1InIelCmXcWsORJd+5yCGOSS3TL2lA+1YXLAoGf47hMm/uT0HvzVhDq7\nD+764ZgRHjN8tpn9N0hz/Gj0zaw+9lOXEXG1DnlGzo016sAOc+2tFZx\n3j8w9cZQJ0zTE2u7Lz8CL9yKXicsOgFhdeyrF4AwtJ2CLtZF383wim2QFi4/Ypkl\nL4lsYnJYnJjQCKgA6bROu0+rA1TUvCzXHbgH7t6eYRcZeKnJNZ+m3PhBs+8W\nov4nLLTz8Q7GN8g6T1//QojS8y ZR9GAr0Z0BbtW8om+fVehPFAMm8x6tS4sTFl0\nUp+i0r4VF7PnvfSIC+AHJUe+a4XPmmphVsnxEpsS+tQ2yUh7Akmt\np8WOECgYAJuaT+FBqIWhvmaymOjUFfQug67+lv7w3qzzWQAq8DyTweFNJ4E\nIbE1RnT86V2xhPr3YgjmRyyONlb/Xr8fZrz8KpmSehT99a+QY6gkIoWrfQ5xS7g6\nI/GDX2x54eANWX0xXKMQXfTU+WN6s5WPl/BL+/Cj43Hfg==\n-----END RSA PRIVATE KEY-----",
"type": "SSHKEYS"
}
}
Warning
Remember, never share this json file because it contains a plain text copy of your private key.
You will need to keep a copy of this file to edit the storage system in the future. To register this system with Tapis, use the following command:
$ tapis systems create -F tacc.work.taccuser.json
+----------------------+------------------------------------------------------+
| Field | Value |
+----------------------+------------------------------------------------------+
| id | tacc.work.taccuser |
| name | Storage system for TACC work directory |
| type | STORAGE |
| default | False |
| available | True |
| description | Storage system for TACC work directory via Stampede2 |
| executionType | None |
| globalDefault | False |
| lastModified | just now |
| maxSystemJobs | None |
| maxSystemJobsPerUser | None |
| owner | taccuser |
| public | False |
| revision | 1 |
| scheduler | None |
| scratchDir | None |
| site | None |
| status | UP |
| uuid | 383424038079107562-242ac112-0001-006 |
| workDir | None |
+----------------------+------------------------------------------------------+
Confirm that it worked by searching for the storage system and listing files in the root directory:
$ tapis systems search --id eq tacc.work.taccuser
+--------------------+----------------------------------------+---------+---------+
| id | name | type | default |
+--------------------+----------------------------------------+---------+---------+
| tacc.work.taccuser | Storage system for TACC work directory | STORAGE | False |
+--------------------+----------------------------------------+---------+---------+
$ tapis files list agave://tacc.work.taccuser/
+-----------+--------------+--------+
| name | lastModified | length |
+-----------+--------------+--------+
| jobs | 2 years ago | 4096 |
| maverick | 2 years ago | 4096 |
| stampede2 | 2 years ago | 4096 |
| wrangler | 2 years ago | 4096 |
+-----------+--------------+--------+
Register an Execution System¶
An execution system contains many of the same fields as a storage system, but it
is a bit more involved. Save the following template for a Stampede2 execution
system into a file called tacc.stampede2.taccuser:
{
"id": "tacc.stampede2.taccuser",
"name": "Execution system for TACC Stampede2",
"description": "Execution system for TACC Stampede2",
"type": "EXECUTION",
"executionType": "HPC",
"scheduler": "SLURM",
"maxSystemJobsPerUser": 50,
"scratchDir": "/scratch/01234/taccuser",
"login": {
"host": "stampede2.tacc.utexas.edu",
"port": 22,
"protocol": "SSH",
"auth": {
"username": "taccuser",
"publicKey": " <enter public key here> ",
"privateKey": " <enter private key here> ",
"type": "SSHKEYS"
}
},
"storage": {
"host": "stampede2.tacc.utexas.edu",
"port": 22,
"protocol": "SFTP",
"rootDir": "/",
"homeDir": "/work/01234/taccuser",
"auth": {
"username": "taccuser",
"publicKey": " <enter public key here> ",
"privateKey": " <enter private key here> ",
"type": "SSHKEYS"
}
},
"queues": [
{
"name": "normal",
"maxProcessorsPerNode": 68,
"maxMemoryPerNode": "96GB",
"maxNodes": 256,
"maxRequestedTime": "48:00:00",
"customDirectives": "-A <enter allocation name here>",
"default": true
}
]
}
Some of the new or changed fields in this execution system include:
type: execution system rather than storage systemexecutionType:HPCindicates a cluster with a job schedulerscheduler: Stampede2 uses a SLURM schedulermaxSystemJobsPerUser: maximum concurrent jobs on the system per userscratchDir: path for job working directory at runtime, relative torootDirlogin: similar to storage, host and credential informationqueues: an array of batch queue definitions for the HPC system
For this execution system, there are two locations to cut and paste your SSH
keys. Again, because keys will be stored in plain text in this file, do not
share this file with anyone and keep it secure. In addition, the queues
parameter has an option called customDirectives which should contain the
name of an allocation you have access to on Stampede2. And finally, as before,
make sure to change the username and paths to match your account on the HPC
system.
Once the appropriate changes have been made to the json file, register the system with Tapis using the following command:
$ tapis systems create -F tacc.stampede2.taccuser.json
+----------------------+---------------------------------------+
| Field | Value |
+----------------------+---------------------------------------+
| id | tacc.stampede2.taccuser |
| name | Execution system for TACC Stampede2 |
| type | EXECUTION |
| default | False |
| available | True |
| description | Execution system for TACC Stampede2 |
| executionType | HPC |
| globalDefault | False |
| lastModified | just now |
| maxSystemJobs | 2147483647 |
| maxSystemJobsPerUser | 50 |
| owner | taccuser |
| public | False |
| revision | 1 |
| scheduler | SLURM |
| scratchDir | /scratch/01234/taccuser/ |
| site | None |
| status | UP |
| uuid | 4903282542648684054-242ac112-0001-006 |
| workDir | |
+----------------------+---------------------------------------+
Finally, confirm that the system exists by searching for it then listing the available queues:
# Search for your private systems
$ tapis systems search --public eq false
+-------------------------+----------------------------------------+-----------+---------+
| id | name | type | default |
+-------------------------+----------------------------------------+-----------+---------+
| tacc.stampede2.taccuser | Execution system for TACC Stampede2 | EXECUTION | False |
| tacc.work.taccuser | Storage system for TACC work directory | STORAGE | False |
+-------------------------+----------------------------------------+-----------+---------+
# List queues on the execution system
$ tapis systems queues list -f json tacc.stampede2.taccuser
[
{
"name": "normal",
"description": null,
"default": true,
"maxUserJobs": -1,
"maxRequestedTime": "48:00:00"
}
]
Additional Help¶
Further information about creating storage and execution systems, including full descriptions of the parameters above as well as other optional parameters, can be found in the Tapis platform documentation