Copy files in Parallel to the Cloud and Beyond!

As an IT professional there might be many times you might be tasked with copying multiple files from one server to another. A simple and straightforward way would be just to issue a single sftp or scp command as follows:

$ scp filename*.dmp remotehost:/Backups/

For the most part that will work fine but if you have a limited network pipe between your source and target server and if you have multiple large files then this serial or single-threaded method may not be the optimal solution especially if time is of the essence because you would end up waiting for each file to be fully copied before the next file would be copied.

I have come up with a cool innovative method I have been using for many years to copy a list of multiple files in parallel from your source server to the target server. Please note this method can be used for just about anything such as multiple export dump files, RMAN backup files, logfiles files  and so on. I would highly recommend this method when copying files to the Cloud or any remote host.

Hope these steps help you to save time and copy your files to the remote server that could be in your own Datacenter, a remote Datacenter or even in the Cloud!

1. First create a file that will contain a list of all of the files to be copied.

$ ls -1 /Backup/*dmp > copy_dump_files   

2. Review the list of the contents of the files.

$ cat copy_dump_files

/Backup/export_01.dmp

/Backup/export_02.dmp

/Backup/export_03.dmp

/Backup/export_04.dmp

/Backup/export_05.dmp

/Backup/export_06.dmp

/Backup/export_07.dmp

/Backup/export_08.dmp

/Backup/export_09.dmp

/Backup/export_10.dmp

/Backup/export_11.dmp

/Backup/export_12.dmp

/Backup/export_13.dmp

/Backup/export_14.dmp

/Backup/export_15.dmp

/Backup/export_16.dmp

/Backup/export_17.dmp

/Backup/export_18.dmp

/Backup/export_19.dmp

/Backup/export_20.dmp

/Backup/export_21.dmp

/Backup/export_22.dmp

3. Edit the file copy_dump_files and on each line of the file put the beginning of each file put the string "scp" and at the end of each line put the string “remotehost:/Backup/” shown as follows:

scp /Backup/export_01.dmp remotest:/Backups/
...


4. Next we split the copy_dump_files file into smaller pieces as follows with only 2 lines per resulting file. Note you could change the number from 2 to anything. The higher the number the more files will be in each split file.

$ split -l 2 copy_dump_files copy_dump_files_1

5. Please note the resulting files will each contain 2 lines as specified in the previous step with the split command.

$ wc -l copy_dump_files_1*
   2 copy_dump_files_1aa
   2 copy_dump_files_1ab
   2 copy_dump_files_1ac
   2 copy_dump_files_1ad
   2 copy_dump_files_1ae
   2 copy_dump_files_1af
   2 copy_dump_files_1ag
   2 copy_dump_files_1ah
   2 copy_dump_files_1ai
   2 copy_dump_files_1aj
   2 copy_dump_files_1ak
  22 total

5. Put all of the copy files into a script shown as follows 

$ ls -1 copy_dump_files_1* > copy_script.sh

6.  Set all of the copy files with executable permissions.

$ chmod 700 copy_*

7. Edit the file copy_script.sh and put the following string at the beginning of each line “nohup ./” and at the end of each line an character “&” which will allow us to run 11 copies simultaneously. Shown as follows:

   $ nohup ./copy_dump_files_1aa  &

   $ nohup ./copy_dump_files_1ab  &

   $ nohup ./copy_dump_files_1ac  &
  ...

  8. Now invoke the copy script to kick off in nohup and in the background to run multiple scp copies to your remote host shown as follows and all copies will run in the background! Please also monitor the target location to ensure the file sizes are increasing.

$ nohup ./copy_script.sh &