File transfer between a dektop and Supercomputers

The basic procedure for moving files to and from supercomputers, for example sequoia, is to move the files first from your desktop to hpcwoods, then from hpcwoods to sequoia (or vice versa if offloading files back to your desktop). All files will be transferred to and from our servers with a program called "secure file transport protocol" or sftp. That said, there are two different ways for you to interface with sftp. These commands are standard linux fare, so they'll look the same in both steps (from your desktop to hpcwoods, and from hpcwoods to sequoia) except that the usernames and destinationhosts will be different, of course. The first is scp (for "Secure Copy") and it works as a single command. From the source machine (here, mimosa) you type a command of the format scp file.ext username@destinationhost:/path/to/file/destination/file.ext

Here's an example: we'll move a file from mimosa to hpcwoods.

mimosa(scp_example)% ls

ga-4-0-8.tar

mimosa(scp_example)% scp ga-4-0-8.tar bwh_TU5P@hpcwoods.olemiss.edu:/home/bwh_TU5P/scp_example_dest/ga-4-0-8.tar

The authenticity of host 'hpcwoods.olemiss.edu (130.74.126.15)' can't be established.

RSA key fingerprint is b7:9b:41:70:f8:11:87:f6:b6:e4:af:6d:00:ad:35:6b.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'hpcwoods.olemiss.edu,130.74.126.15' (RSA) to the list of known hosts.

Password:

ga-4-0-8.tar 100% 9210KB 9.0MB/s 00:01

mimosa(scp_example)%

You'll only get that "Are you sure..." prompt the first time. Enter yes, and then enter your password at the Password: prompt, and the transfer will complete. For all subsequent connections, you'll just get the password prompt. The password you want here is the one for the destination machine (here, hpcwoods). Most of the flags (ie, options) for scp mimic those for regular old cp. For example, the most useful flag to scp is -r, which stands for "recursive" and is used to copy whole directories. Here, ga-4-0-8 tarball has been unpacked and the resulting directory will be copied in its entirety.

Ordinary scp doesn't like that:

mimosa(scp_example)% scp ga-4-0-8 bwh_TU5P@hpcwoods.olemiss.edu:/home/bwh_TU5P/scp_example_dest/ga-4-0-8

Password:

ga-4-0-8: not a regular file

But with the -r flag, the transfer works:

mimosa(scp_example)% scp -r ga-4-0-8 bwh_TU5P@hpcwoods.olemiss.edu:/home/bwh_TU5P/scp_example_dest/

Password:

INSTALL 100% 585 0.6KB/s 00:00

NOTES 100% 8620 8.4KB/s 00:00

README 100% 15KB 15.5KB/s 00:00

And so on (it actually spills out a progress bar for every individual file in the directory). When it's done there's a directory on hpcwoods that's exactly like the source directory on mimosa:

hpcwoods(~/scp_example_dest)% ls

ga-4-0-8 ga-4-0-8.tar

hpcwoods(~/scp_example_dest)% ls ga-4-0-8/

armci config global INSTALL LinAlg NOTES README tcgmsg-mpi

cca ga++ GNUmakefile license.txt ma pario tcgmsg

Note that the tarball ga-4-0-8.tar came from the first transfer (where we copied the tarball) and the subdir ga-4-0-8 and all of its files subdirs came from the second "recursive" transfer. Note also that this is secure copy, not secure move; the original files on mimosa are still there after the transfer.

The second interface is called sftp. It's neither better nor worse, but is very different. With sftp, you're going to initiate a connection to the destination machine that's going to remain open, so you can navigate around and whatnot. It works like this. Let's move the same file as before, ga-4-0-8.tar.

mimosa(sftp_example)% cd /ptmp/bwhopkin/sftp_example/

Directory: /ptmp/bwhopkin/sftp_example

mimosa(sftp_example)% ls

ga-4-0-8 ga-4-0-8.tar

mimosa(sftp_example)% sftp bwh_TU5P@hpcwoods.olemiss.edu

Connecting to hpcwoods.olemiss.edu...

Password:

sftp> pwd

Remote working directory: /home/bwh_TU5P

sftp> ls

bin file.xml

grad_medium.com molpro-mpp-2008.1-5.Linux_em64t.sh.gz

pierre scp_example_dest

sftp_example_dest testfile.com

textfile.txt

sftp> cd sftp_example_dest

sftp> ls

sftp> put ga-4-0-8.tar

Uploading ga-4-0-8.tar to /home/bwh_TU5P/sftp_example_dest/ga-4-0-8.tar

ga-4-0-8.tar 100% 9210KB 9.0MB/s 00:00

sftp> ls

ga-4-0-8.tar

sftp> exit

So here what we did was open the sftp connection, which gave us an sftp command prompt. At that prompt we could do stuff on the remote machine: pwd, ls, cd, and whatnot. To transfer the file, we used the "put" command. This tells sftp to get the relevant file from the source machine and put it in the current directory on the destination machine. There's also a "get" command, which tells sftp to get a file with the relevant name from the destination machine and put it on the source machine.

With both interfaces you can do stuff like using wildcard-characters to move lists of files:

mimosa(symm_recog)% scp c2v.* bwh_TU5P@hpcwoods.olemiss.edu:/home/bwh_TU5P/scp_example_dest/

Password:

c2v.com 100% 130 0.1KB/s 00:00

c2v.log 100% 1579 1.5KB/s 00:00

c2v.LOG 100% 1579 1.5KB/s 00:00

c2v.out 100% 37KB 36.9KB/s 00:00

c2v.OUT 100% 4886 4.8KB/s 00:00

c2v.pbs 100% 2303 2.3KB/s 00:00

mimosa(symm_recog)%