UNIX Tools: fdupes – Get rid of duplicate files

fdupes is a  tool for identifying or deleting duplicate files residing within specified directories. It first compares file sizes and MD5 signatures, and then performs a byte-by-byte check for verification.

1. Install fdupes

Type the following commands to install fdupes in FreeBSD:

# cd /usr/ports/sysutils/fdupes
# make install clean

Type the following command to install fdupes in CentOS / Fedora / RHEL (make sure you have rpmforge repo enabled):

# yum install fdupes

Type the following command to install fdupes in Debian / Ubuntu Linux:

# apt-get install fdupes

2. Using fdupes for finding and deleting duplicates

To find duplicate files in /etc/ directory, run :

# fdupes /etc

Sample outputs:

/etc/profile
/etc/profile2

You can recursively search every directory given follow sub-directories encountered within the -r option:

# fdupes -r /etc

You can also find duplicates files in two or more directories:

# fdupes /etc /usr/local/etc

OR

# fdupes -r /etc /usr/local/etc /home/user/etc

Delete unwanted files
You can force fdupes to prompt you for files to preserve, deleting all others:

# fdupes -d /etc

Sample outputs:

[1] /etc/profile
[2] /etc/profile2

Set 1 of 1, preserve files [1 - 2, all]: 1

[+] /etc/profile
[-] /etc/profile2

You can see the size of the duplicate files with the -S option:

# fdupes -S /etc

Sample outputs:

1029 bytes each:
/etc/profile
/etc/profile2