File transfer between a dektop and Supercomputers
The basic procedure for moving files to and from supercomputers, for example
sequoia, is to move the files first from your desktop to hpcwoods, then from
versa if offloading files back to your desktop). All files will be transferred
to and from our servers with a program called "secure file transport protocol" or
sftp. That said, there are two different ways for you to interface with sftp.
These commands are standard linux fare, so they'll look the same in both steps
(from your desktop to hpcwoods, and from hpcwoods to sequoia) except that the
usernames and destinationhosts will be different, of course.
The first is scp (for "Secure Copy") and it works as a single command. From
the source machine (here, mimosa) you type a command of the format
scp file.ext username@destinationhost:/path/to/file/destination/file.ext
Here's an example: we'll move a file from mimosa to hpcwoods.
mimosa(scp_example)% scp ga-4-0-8.tar bwh_TU5P@hpcwoods.olemiss.edu:/home/bwh_TU5P/scp_example_dest/ga-4-0-8.tar
The authenticity of host 'hpcwoods.olemiss.edu (188.8.131.52)' can't be established.
RSA key fingerprint is b7:9b:41:70:f8:11:87:f6:b6:e4:af:6d:00:ad:35:6b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'hpcwoods.olemiss.edu,184.108.40.206' (RSA) to the list of known hosts.
ga-4-0-8.tar 100% 9210KB 9.0MB/s 00:01
You'll only get that "Are you sure..." prompt the first time. Enter yes,
and then enter your password at the Password: prompt, and the transfer will
complete. For all subsequent connections, you'll just get the password prompt.
The password you want here is the one for the destination machine (here, hpcwoods).
Most of the flags (ie, options) for scp mimic those for regular old cp. For
example, the most useful flag to scp is -r, which stands for "recursive" and
is used to copy whole directories. Here, ga-4-0-8 tarball
has been unpacked and the resulting directory will be copied
in its entirety.
Ordinary scp doesn't like that:
mimosa(scp_example)% scp ga-4-0-8 bwh_TU5P@hpcwoods.olemiss.edu:/home/bwh_TU5P/scp_example_dest/ga-4-0-8
ga-4-0-8: not a regular file
But with the -r flag, the transfer works:
mimosa(scp_example)% scp -r ga-4-0-8 bwh_TU5P@hpcwoods.olemiss.edu:/home/bwh_TU5P/scp_example_dest/
INSTALL 100% 585 0.6KB/s 00:00
NOTES 100% 8620 8.4KB/s 00:00
README 100% 15KB 15.5KB/s 00:00
And so on (it actually spills out a progress bar for every individual file in the directory). When it's done there's a directory on hpcwoods that's exactly like the source directory on mimosa:
hpcwoods(~/scp_example_dest)% ls ga-4-0-8/
armci config global INSTALL LinAlg NOTES README tcgmsg-mpi
cca ga++ GNUmakefile license.txt ma pario tcgmsg
Note that the tarball ga-4-0-8.tar came from the first transfer (where we copied the tarball) and the subdir ga-4-0-8 and all of its files subdirs came from the second "recursive" transfer. Note also that this is secure copy, not secure move; the original files on mimosa are still there after the transfer.
The second interface is called sftp. It's neither better nor worse, but is very different. With sftp, you're going to initiate a connection to the destination machine that's going to remain open, so you can navigate around and whatnot. It works like this. Let's move the same file as before, ga-4-0-8.tar.
mimosa(sftp_example)% cd /ptmp/bwhopkin/sftp_example/
mimosa(sftp_example)% sftp bwh_TU5P@hpcwoods.olemiss.edu
Connecting to hpcwoods.olemiss.edu...
Remote working directory: /home/bwh_TU5P
sftp> cd sftp_example_dest
sftp> put ga-4-0-8.tar
Uploading ga-4-0-8.tar to /home/bwh_TU5P/sftp_example_dest/ga-4-0-8.tar
ga-4-0-8.tar 100% 9210KB 9.0MB/s 00:00
So here what we did was open the sftp connection, which gave us an sftp command prompt. At that prompt we could do stuff on the remote machine: pwd, ls, cd, and whatnot. To transfer the file, we used the "put" command. This tells sftp to get the relevant file from the source machine and put it in the current directory on the destination machine. There's also a "get" command, which tells sftp to get a file with the relevant name from the destination machine and put it on the source machine.
With both interfaces you can do stuff like using wildcard-characters to move lists of files:
mimosa(symm_recog)% scp c2v.* bwh_TU5P@hpcwoods.olemiss.edu:/home/bwh_TU5P/scp_example_dest/
c2v.com 100% 130 0.1KB/s 00:00
c2v.log 100% 1579 1.5KB/s 00:00
c2v.LOG 100% 1579 1.5KB/s 00:00
c2v.out 100% 37KB 36.9KB/s 00:00
c2v.OUT 100% 4886 4.8KB/s 00:00
c2v.pbs 100% 2303 2.3KB/s 00:00