|
|
This page provides an introduction to moving data around Gaea and PPAN, as well as transferring data to a local laptop or workstation.
|
|
This page provides an introduction to moving data around Gaea and PPAN, as well as transferring data to a local laptop or workstation.
|
|
|
|
|
|
|
|
|
### f5<->f6
|
|
|
|
At this moment, f5 (f6) cannot directly access f6 (f5). Instead, users will need to transfer data through LDTN or RDTN nodes. Below is a quick example of how to submit an interaction job and transfer data between F5 and F6:
|
|
|
|
|
|
|
|
```
|
|
|
|
Yi-cheng.Teng@gaea55:~> salloc --partition=ldtn_c5
|
|
|
|
Yi-cheng.Teng@dtn57:~>module load gcp
|
|
|
|
Yi-cheng.Teng@dtn69:~> gcp -r /gpfs/f5/cefi/world-shared/nep_xml/ /gpfs/f6/ira-cefi/world-shared/upload/
|
|
|
|
Yi-cheng.Teng@dtn69:~> ls /gpfs/f6/ira-cefi/world-shared/upload/
|
|
|
|
nep_xml
|
|
|
|
```
|
|
|
|
Keep in mind that the interactive method only works on C5 currently. If users want to transfer data between F5 and F6 on C6, they will need to submit a job through the Slurm Queueing System. Below is an example script:
|
|
|
|
```
|
|
|
|
#!/usr/bin/sh -e
|
|
|
|
#SBATCH --cluster=es
|
|
|
|
#SBATCH --partition=ldtn_c5
|
|
|
|
#SBATCH --nodes=1
|
|
|
|
#SBATCH --time=04:00:00
|
|
|
|
|
|
|
|
# How to use:
|
|
|
|
# from c6 login node:
|
|
|
|
#sbatch f56.sh /gpfs/f5/... /gpfs/f6/...
|
|
|
|
|
|
|
|
# Check if the correct number of arguments is provided
|
|
|
|
if [ "$#" -ne 2 ]; then
|
|
|
|
echo "Usage: $0 <source_directory> <destination_directory>"
|
|
|
|
exit 1
|
|
|
|
fi
|
|
|
|
|
|
|
|
# Assign command-line arguments to variables
|
|
|
|
source_directory="$1" #like /gpfs/f6/cefi/scratch/Yi-cheng.Teng/github/ci_data/nwa12_input
|
|
|
|
destination_directory="$2" #like /gpfs/f6/ira-cefi/scratch/Yi-cheng.Teng/
|
|
|
|
|
|
|
|
/ncrc/usw/gcp/local/opt/gcp/2.3.33/gcp -cd -r gaea:"$source_directory" "$destination_directory"
|
|
|
|
|
|
|
|
let cp_status=$?+256
|
|
|
|
if [ $cp_status -ne 256 ]
|
|
|
|
then
|
|
|
|
# An exit status greater than 256 will indicate the copy
|
|
|
|
# failed. Any exit status less than 256 will be a failure
|
|
|
|
# from something else
|
|
|
|
exit $cp_status
|
|
|
|
fi
|
|
|
|
```
|
|
|
|
Keep in mind that you will need to load the gcp module before submitting the script.
|
|
|
|
|
|
|
|
### Gaea <-> GFDL
|
|
|
|
Users can transfer data between GFDL and Gaea filesystems with GCP. This can be done on the login nodes and rdtn’s only. Users can interactively run `gcp` commands from a login node or submit `gcp` calls in scripts to run in the `rdtn` queue.
|
|
|
|
```
|
|
|
|
module load gcp
|
|
|
|
gcp gaea:/gpfs/f5/<project>/scratch/$USER/file gfdl:/gfdl/specific/path/file
|
|
|
|
gcp gfdl:/gfdl/specific/path/file gaea:/gpfs/f5/<project>/scratch/$USER/path/file
|
|
|
|
```
|
|
|
|
|
|
|
|
### PPAN <-> External (Untrusted) Data Transfers
|
|
|
|
To transfer data from PPAN/archive to your local laptop or computer, you can use Globus.. First, install [Globus Connect Persona](https://www.globus.org/globus-connect-personal) and create an endpoint on your local device. Once set up, use the [Globus](https://app.globus.org/) web interface to transfer data between NOAA RDHPCS UDTN (e.g., on PPAN, the transferred data will be placed under `/collab1/data_untrusted/$USER`) and your local computer. |
|
|
|
\ No newline at end of file |
