Top 5 Tools to Find and Remove Duplicate Files in Linux

0

Managing files is a complicated task in itself. Add to that large volume of duplicate files which usually hog storage space, and the process becomes more and more difficult.

While the standard way to deal with duplicate files is to locate and delete them manually. However, using a dedicated duplicate file finder can significantly speed up the process.

So, if you are planning to get rid of duplicate files and clean up your computer, here is a list of some of the best tools to find and remove duplicate files in Linux.

fslint duplicate file remover

Fslint is a GUI and command line interface based utility to clean various types of clutter from your system. It calls this clutter “lint” and offers several tools to help you perform a multitude of tasks, including finding duplicate files, empty directories, and problematic file names.

By providing both graphical and command-line modes of operation, fslint makes it easier for new Linux users to free their computer’s storage from all kinds of system fluff.

To access fslint through the GUI, all you need to do is open the terminal and run the fslint-gui order.

As for the advanced features, the program offers 10 different features in CLI mode such as findup, findu8, findnl, findtf and found. By using them, you can narrow down the search results to increase your chances of finding specific types of duplicate files on your system.

How to install fslint

On Debian based distributions like Ubuntu:

sudo apt install fslint

On RHEL-based distributions like CentOS and Fedora:

sudo yum install fslint
sudo dnf install fslint

On Arch Linux and Manjaro:

sudo pacman -S fslint

fdupes under linux

Fdupes is one of the easiest programs to identify and remove duplicate files residing in directories. Released under the MIT license on GitHub, it is free and open-source.

The program works by using the md5sum signature and the byte-by-byte comparison check to determine duplicate files in a directory. If needed, you can also perform recursive searches, filter search results, and get a summary view of duplicate files discovered.

Once you identify the duplicate files in a directory, you can then use fdupes to remove the files or replace them with links to the original file.

Installation of Fdupes

On Debian-based distributions:

sudo apt install fdupes

On RHEL-based distributions:

sudo yum install fdupes
sudo dnf install fdupes

To install on Arch Linux and Manjaro:

sudo pacman -S fdupes

Related: How to Find and Remove Duplicate Files on Linux Using fdupes

use rdfind to remove duplicate files

Rdfind is another Linux utility to help you find redundant files on your computer in different directories. It relies on comparing files based on content – not name – to identify duplicates, which makes it more efficient at its job.

To achieve this, the program works by classifying equal files in a directory and determining the original and duplicates: the highest-rated file is selected as the original while the others are duplicates.

Additionally, rdfind can also calculate checksums to compare files if needed. And the best part is that it saves the scanned results in a results.txt file in the home directory, so you can refer to it when you’re about to remove duplicates to make sure you don’t remove the bad ones.

Of course, as with most other duplicate file finder tools, rdfind also offers preprocessors to sort files, ignore empty files, or set symbolic links. Last but not the least, there is also an option to remove duplicate files.

Related: What is a Symbolic Link (Symbolic Link)? How to create one in Linux

How to install rdfind

On Debian / Ubuntu:

sudo apt install rdfind

On Fedora / CentOS:

sudo dnf install rdfind

dupeguru running under linux

DupeGuru is a cross-platform tool for finding and removing duplicate files on your computer. One of its best features is the ability to customize the match engine to suit your preferences to increase your chances of finding the right type of duplicate files in a directory. And similar to a few other duplicate finder programs, it also offers a graphical interface for easy operations.

Speaking of functionality, dupeGuru uses its fuzzy match algorithm to analyze file names or file contents and find duplicates quickly and efficiently.

Moreover, it is also good at handling music and picture specific information, giving it an edge over other duplicate file finder tools. Additionally, if needed, you have the option of modifying its matching engine to locate exactly what type of duplicate files you want to eliminate.

DupeGuru also allows you to remove duplicate files. And for that, it has a reference directory system, which prevents you from accidentally deleting bad files. Besides deleting, there is also the option to move or copy them elsewhere.

Installing DupeGuru

On Debian-based distributions:

sudo add-apt-repository ppa:dupeguru/ppa
sudo apt-get update
sudo apt-get install dupeguru

On ArchLinux:

sudo pacman -S dupeguru

rmlint on linux

Rmlint is another lint, not just duplicate, find and removal tool for Linux. It is free and extremely fast to identify duplicate files and directories on your system. You also get support for the Btrfs storage format, which sets it apart from other tools on this list.

Speaking of, some of the other aspects where rmlint trumps other competing duplicate file removal tools include the ability to search for files based on a particular time period, find files with user IDs / group broken and find unstripped binaries that take up a lot of space. Besides, similar to a few other programs, it also saves the scanned results in rmlint.json and rmlint.sh files, which are useful during the delete operation.

However, note that unlike other tools, rmlint is not the easiest to use – it generates a script to remove duplicates, which requires a certain level of understanding to be used effectively.

How to install rmlint

On Debian-based distributions:

sudo apt install rmlint

On Fedora and CentOS:

sudo yum install rmlint
sudo dnf install rmlint

On Arch-based distributions like Manjaro:

sudo pacman -S rmlint

Keep duplicate files remote in Linux

Using the duplicate file finder programs listed above, you can easily identify duplicate files that might be taking up space on your computer and remove them completely. However, a tip when working with such tools is to be very careful in your actions to avoid ending up deleting important files and documents on your system.

In case you are a little skeptical about which files to delete and which to keep, be sure to back up all data on your system for added security.


Linux Backup Tools
The 8 best file backup apps for Linux

Regular data backups are crucial, whether it is a server or a local machine. Check out these eight file backup apps for Linux.

Read more


About the Author


Source link

Leave A Reply

Your email address will not be published.