Linux, FreeBSD, Juniper, Cisco / Network security articles and troubleshooting guides

It is currently Wed Jan 18, 2017 7:40 pm

Tutorials applicable on more than one Unix/Linux OS and shell scripts: ssh / openssl / protocols.

Author Message
Post  Post subject: Rsync: How to transfer and remove the source files afterwards  |  Posted: Thu Jan 22, 2015 2:55 pm

Joined: Tue Aug 04, 2009 9:16 am
Posts: 245


Rsync: How to transfer and remove the source files afterwards

Example how to move files with RSYNC

This articles is valid for all Unix flavours that have rsync available.

Rsync is one powerful tool for Linux administrators that allows one to synchronise two directories/paths based on source/destination model.

Among the many features it offers, there is one particularly useful when a migration (not backup) has to be performed and the source directory has to be wiped.

The rsync option that allows for source files and directories to be deleted is “--remove-source-files“.

But this is not enough for successful migration with rsync. In many situations, the owner/permissions and modification times of the source files and directories should be preserved on the destination directories. Here are a few options to accomplish this:
$ rsync 2>&1 | grep preserv
-H, --hard-links            preserve hard links
-p, --perms                 preserve permissions
     --executability         preserve the file's executability
-o, --owner                 preserve owner (super-user only)
-g, --group                 preserve group
     --devices               preserve device files (super-user only)
     --specials              preserve special files
-t, --times                 preserve times
-O, --omit-dir-times        omit directories when preserving times

Another set of rsync options offer recursively (to migrate directories instead of files), copy symlinks, preserve group and the “-a” option that combines all the above rsync options.

rsync 2>&1 | grep -E '^ \-(a|r|l|p|t|g|o|D)'
-a, --archive               archive mode; same as -rlptgoD (no -H)
-r, --recursive             recurse into directories
-l, --links                 copy symlinks as symlinks
-p, --perms                 preserve permissions
-o, --owner                 preserve owner (super-user only)
-g, --group                 preserve group
-D                          same as --devices --specials
-t, --times                 preserve times
$ rsync 2>&1 | grep -E remove-source
     --remove-source-files   sender removes synchronized files (non-dirs)

Delete source files/directories after rsync transfer is complete:

Before I provide the example, I should provide the environment details. Below, I will create a source directory having 4 sub directories and one file in each subdirectory. Then rsync will be used together with “—remove-source-files” and then I will show how the source and destination structure looks like.
acBook-Pro-van-Andrei:~ Andrei$ mkdir -p source/{1,2,3,4}
my-mac:~ Andrei$ touch source/{1,2,3,4}/files
my-mac:~ Andrei$ find source/ -type f
my-mac:~ Andrei$ mkdir destination
my-mac:~ Andrei$ rsync -rav --remove-source-files source/ destination/
building file list ... done

sent 359 bytes  received 138 bytes  994.00 bytes/sec
total size is 0  speedup is 0.00
my-mac:~ Andrei$ find source/ -ls
1173801        0 drwxr-xr-x    6 Andrei           staff                 204 Jan 22 19:42 source/
1173802        0 drwxr-xr-x    2 Andrei           staff                  68 Jan 22 19:43 source//1
1173803        0 drwxr-xr-x    2 Andrei           staff                  68 Jan 22 19:43 source//2
1173804        0 drwxr-xr-x    2 Andrei           staff                  68 Jan 22 19:43 source//3
1173805        0 drwxr-xr-x    2 Andrei           staff                  68 Jan 22 19:43 source//4
my-mac:~ Andrei$ find destination/ -ls
1177476        0 drwxr-xr-x    6 Andrei           staff                 204 Jan 22 19:42 destination/
1183980        0 drwxr-xr-x    3 Andrei           staff                 102 Jan 22 19:42 destination//1
1183984        0 -rw-r--r--    1 Andrei           staff                   0 Jan 22 19:42 destination//1/files
1183981        0 drwxr-xr-x    3 Andrei           staff                 102 Jan 22 19:42 destination//2
1183985        0 -rw-r--r--    1 Andrei           staff                   0 Jan 22 19:42 destination//2/files
1183982        0 drwxr-xr-x    3 Andrei           staff                 102 Jan 22 19:42 destination//3
1183986        0 -rw-r--r--    1 Andrei           staff                   0 Jan 22 19:42 destination//3/files
1183983        0 drwxr-xr-x    3 Andrei           staff                 102 Jan 22 19:42 destination//4
1183987        0 -rw-r--r--    1 Andrei           staff                   0 Jan 22 19:42 destination//4/files

Let’s explain the above output: After rsync is used, only source files are deleted (not directories). This is described in rsync help. Next, the find commands show that the source directory has only the four subdirectories. The destination directory includes the 4 subdirectories as well as the files that were synchronised / migrated.

Few notes:
- Before running this live, these options should be tested in advance.
- Other rsync options like the below ones should be studies and not used unless really necessary:
-n, --dry-run               show what would have been transferred
     --delete                delete extraneous files from destination dirs
     --delete-before         receiver deletes before transfer (default)
     --delete-during         receiver deletes during transfer, not before
     --delete-after          receiver deletes after transfer, not before
     --delete-excluded       also delete excluded files from destination dirs

If the rsync transfer is performed over the network, another option is useful to speed up the transfer:
-z, --compress              compress file data during the transfer

And if the transfer speeds are required:
     --progress              show progress during transfer

Display posts from previous:  Sort by  
E-mail friendPrint view
Who is online
Users browsing this forum: No registered users and 0 guests
You can post new topics in this forum
You can reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum
Jump to:  
News News Site map Site map SitemapIndex SitemapIndex RSS Feed RSS Feed Channel list Channel list

Delete all board cookies | The team | All times are UTC - 5 hours [ DST ]