UNIX Tools: fdupes – Get rid of duplicate files
fdupes is a tool for identifying or deleting duplicate files residing within specified directories. It first compares file sizes and MD5 signatures, and then performs a byte-by-byte check for verification.
1. Install fdupes
Type the following commands to install fdupes in FreeBSD:
# cd /usr/ports/sysutils/fdupes
# make install clean
Type the following command to install fdupes in CentOS / Fedora / RHEL (make sure you have rpmforge repo enabled):
# yum install fdupes
Type the following command to install fdupes in Debian / Ubuntu Linux:
# apt-get install fdupes
2. Using fdupes for finding and deleting duplicates
To find duplicate files in /etc/ directory, run :
# fdupes /etc
Sample outputs:
/etc/profile
/etc/profile2
You can recursively search every directory given follow sub-directories encountered within the -r option:
# fdupes -r /etc
You can also find duplicates files in two or more directories:
# fdupes /etc /usr/local/etc
OR
# fdupes -r /etc /usr/local/etc /home/user/etc
Delete unwanted files
You can force fdupes to prompt you for files to preserve, deleting all others:
# fdupes -d /etc
Sample outputs:
[1] /etc/profile
[2] /etc/profile2
Set 1 of 1, preserve files [1 - 2, all]: 1
[+] /etc/profile
[-] /etc/profile2
You can see the size of the duplicate files with the -S option:
# fdupes -S /etc
Sample outputs:
1029 bytes each:
/etc/profile
/etc/profile2