PC Applications

General discussion

How can I get rid of all my duplicate files in multiple hard drives?

by Lee Koo (ADMIN) CNET staff/forum admin / February 25, 2011 6:44 AM PST
Question:

How can I get rid of all my duplicate files in multiple hard drives?


Is there a program that will go through both internal and external hard drives to display those files that are duplicated, triplicated, etc.? Unfortunately, I now have several copies of computer backups on my current computer due to former computer issues on other computers. I did not have time to go through all the documents to see if one system had documents updated within files that another may not have. I believe I have three other backups of my former computers on my system now, which does use a lot of memory, but it is also not an efficient use of storage. Now, I have an external hard drive that has backed up all documents and pictures on its hard drive, which means it, too, has the additional backups on it. I want to "clean house"! I'm running a Windows Vista Home system. Thank you for your assistance.

--Submitted by Cynthia S.

If you have any recommendations or tips for Cynthia, please click on the reply link below and submit away. Please be as detailed as possible when providing a how-to solution and if referencing a utility, please include a link to the product. Thank you!
Discussion is locked
You are posting a reply to: How can I get rid of all my duplicate files in multiple hard drives?
The posting of advertisements, profanity, or personal attacks is prohibited. Please refer to our CNET Forums policies for details. All submitted content is subject to our Terms of Use.
Track this discussion and email me when there are updates

If you're asking for technical help, please be sure to include all your system info, including operating system, model number, and any other specifics related to the problem. Also please exercise your best judgment when posting in the forums--revealing personal information such as your e-mail address, telephone number, and address is not recommended.

You are reporting the following post: How can I get rid of all my duplicate files in multiple hard drives?
This post has been flagged and will be reviewed by our staff. Thank you for helping us maintain CNET's great community.
Sorry, there was a problem flagging this post. Please try again now or at a later time.
If you believe this post is offensive or violates the CNET Forums' Usage policies, you can report it below (this will not automatically remove the post). Once reported, our moderators will be notified and the post will be reviewed.
Collapse -
Glary Utilities
by H3710 / February 25, 2011 8:48 AM PST

Try Glary Utilities (http://www.glaryutilities.com/). There's a free version that will do a duplicate (triplicate, whatever) file search and much, much more. Another fantastic, inexpensive program is Scooter Software's Beyond Compare (http://www.scootersoftware.com/moreinfo.php). It will cost you $30, but is well worth the price if you want to keep your hard drive and backup drives free of clutter.

Collapse -
question
by wtstandup / March 4, 2011 12:09 PM PST
In reply to: Glary Utilities

I have a program that will show my duplicate files and I can read they're the same files it finds.

My question is if there is 2 or more files of anything is it safe to delete all but the one file? Getting rid of the duplicated files.

Wayne

Collapse -
depends on what the file is
by H3710 / March 7, 2011 3:26 AM PST
In reply to: question

for instance, if it's a .dll, I would not delete it, since several different programs might use the same .dll and each program will look for it in its own directory. But if it's a .jpg, bitmap or wave file -- a photo you've taken or a song you've downloaded that's showing up in different folders, it would certainly be safe to delete the duplicate (triplicate) copies that sometimes end up in various places on your computer. What I frequently do is list the files according to size, then delete the largest files that I know are safe to eliminate.

Collapse -
Deleting duplicate files
by nsheff / March 7, 2011 5:21 AM PST
In reply to: question

The answer is yes. You can delete all the duplicates you find leaving only the original file. Pictures and music are not too much of a problem, but beware of .dll and other operating files. Sometimes different programs use the same name for such files which are needed by the opereating program. Usually, the program knows where to search (in its own folder), so it is possible that deleting such files might make the file unlocateable.

There is also a very good program for music dupes called oddly enough DMFF Duplicate Music File Finder which allows you to search across many drives and folders and will show the name and duration of each music file. Good luck.

Collapse -
Duplicate Finder
by wersan / October 24, 2011 2:10 AM PDT
In reply to: Glary Utilities

You
can try Duplicate Finder. It will help you find duplicate images (photos,
pictures). Search by contents - files may have different names, but similar
content. You can also use filters to search for duplicates of a certain type
only. Try it.
http://www.duplicate-finder-download.com

Collapse -
Duplicates files on multiple hard drives

Try using Glary Utilities. It will find all your duplicate files, list them, and give you the choice as to what you would like to do. It works really well, and it is free for the basic program.

Collapse -
Glary Utillities
by strooierman / March 4, 2011 5:19 PM PST

Hi Wayne,
I have just used glary and it found 122 duplicate files, silly question but the files I see are they just the duplicates (it shows at least 2 per file) or are they the ones I can delete?
Dave

Collapse -
Are they just the duplicates?
by Gerdd / March 5, 2011 2:55 PM PST
In reply to: Glary Utillities

No, they are all the instances you have. Since your program is too dumb to tell the difference you have to decide now, which is the original you want to keep and which is the duplicate. SO don't erase them all or you'll go from haing two of everything to having zero. In which case I hope you have decent backups (another form of duplication, but quite welcome at times ...)

Collapse -
Update
by wtstandup / March 4, 2011 8:52 PM PST

I down loaded glary utilities because you suggested and after I did, and it only did what I already have/2 different programs that do that, as I first stated that will find duplicate files. But my only question is what is safe to delete or not safe delete!
When I realized I had just downloaded a program I don't need and already have better. I came back to reread your thread and I see. Where you never did answer my question you went off on something else, pushing glary utilities. So that is my fault missing you never answered my question and ended up wasting a lot of my time!
In that now I have another program running I don't need at all, slowing my PC up and now I'm dumping the program/glary's. I'd suggest to you to keep others from wasting time when they/I am trying to have a question answered, if you don't have the answer "to the actual question" that you, just pass on the question. I do see others are selling there chosen products too, like you and like you they're not answering my question either!

Collapse -
Think before ... [you fill in the blank]
by Gerdd / March 5, 2011 2:47 PM PST
In reply to: Update

There is no software yet that I know of that tells you what to do with the duplicate files it finds. There must be hundreds of programs that find duplicate files for you. Some just report, others give you an option to tick the ones you want deleted, and so on.

So, before you even go and find duplicates - on external drives, in backup images etc, it can be a time consuming job - think of a strategy: As we all know, it is reckless to have only one copy of your data.

But do you just want one backup of your current files? Two copies in two different locations are better. What about historical backups?

Consider this: Somehow you manage to lose a file. It is important but you don't access it all the time. You make a backup and discard the old one. The new backup doesn't have the file - you deleted it before you took the backup. If you didn't keep some historical backups (archives) around your file is now gone for good. Same thing if you didn't lose it but it got infected by some virus. If you back up the infected file over the last or only good copy - end of story.

So, having all kinds of historical backups around is not all bad. And the $30 to buy a duplicate finder program could also buy you another external drive to back things up to once again and get some extra redundancy.

No - you don't want just one backup of everything any more than you want the mess of duplicate copies of ancient backups. You want an archive of one backup every so many months, and a second copy of that at your friend's house (have it encrypted if you don't trust him/her that much ...)

I just started exactly that - took a 2TB external drive and collected all backups on it, compared them a bit, also compared what I found on old drives in the grab bag and of course what was online on the drives in my active computers. It is - as you will doubtlessly find out - a time consuming job, but it will get better once the foundation is laid. Then, once you have all the backups you want, make the duplicate copy and then go back and eliminate all the spurious copies you found all over the place. You may get a lot of the disk space back that you invested into your archive, but that is the last thing that should worry you at the current prices.

The main reason I am doing this is so that I know what I have and where - for a few years now I could start a hunt for a file in many locations. I usually would find it, but there was no method, no "algorithm" to the search, and thus no guarantee of success.

Did you notice that I didn't even start discussing what program to use? That is because there is no program that will do for you automatically what I just described. And for the rather mundane task of finding files on your drives and comparing them by name, timestamp and contents most of the available programs will do an equally adequate job.

Collapse -
good advice
by strooierman / March 5, 2011 8:13 PM PST

thanks gerdd sound advice.

Collapse -
Just pass
by rebride / March 6, 2011 4:27 PM PST
In reply to: Update

I would think if they don't have the answer "to the actual question" that 'you' just pass on their answer.

Collapse -
duplicates
by guesswho69 / April 27, 2011 5:02 AM PDT
In reply to: Update

And your better program is???

Collapse -
Many programs will do the job

I use several programs which will do the job. One is Total Commander for general uses and can compare files in different directories so that you can determine if they are identical or have different dates but the same size etc. Also, there is a utility called JV16 that allows you to do many things, one of which is to identify duplicates. The thing is that you can determine the level of similarity, like some files were pictures which you renamed without deleting the original file. So you may need to determine which program will allow you to detect files across folders; JV16 seems to be very good at that as long as I use a low enough folder to allow it to search across sub-folders, which are higher level. There are also several programs that will do this with specific programs, like Marlelsoft dup eliminator for iTunes

Collapse -
Auslogics Duplicate File Finder or Easy Duplicate File Finde

I have used these program with great success: Easy Duplicate File Finder 3.1.1.41
Auslogics Duplicate File Finder 2.0.6.55. The great thing is you can download them right her on CNET and Auslogics is FREE!

Collapse -
Auslogics duplicate file finder.

I have used Auslogics Dup file finder but it only oicks out music and videos even though I clicked on the all files sign Easy dup 'found over 1800 dup files even though I haven't got that many on my computer and I was trying to get rid of duplicates in a folder of 800 poetry files .
Wyn Windows 7

Collapse -
How can I get rid of all my duplicate files in multiple hard
by nlesser / February 25, 2011 9:07 AM PST

AVG PC Tune-Up 2011 has a very effective Duplicate File Finder among its utilities. This can be obtained at http://www.avg.com/us-en/avg-pctuneup. I was able to remove nearly 20% of document file space.

Collapse -
AVG PC Tune-Up 2011 is my favorite
by figmentKLM / February 25, 2011 3:57 PM PST

I use this periodically. It makes it easy since it lists them side by side. Just click and delete. It also has dozens of other tuneup tools. I routinely put this on customers computers. Well worth the $30

Collapse -
Comodo System Cleaner has a built in duplicate files finder.

Comodo System Cleaner v 2.2.335611.5 (old version) has the ability to search for duplicate files stoed on multiple drives. It is available to download from:

Download Locations:

Setup for Windows Vista 32-bit/64-bit, Windows XP 32-bit/64-bit, Windows 7 32-bit/64-bit

http://system-cleaner.comodo.com/

Or the portable version from:

http://download.Comodo.com/csc/download/setups/CSC_SETUP_2.2.335611.5_xp_vista_server2003

Once installed simply run the program then select the Disk Cleaner tab followed by the Duplicate Files tab then select any or all of the available drives to scan. It will take quite some time to complete the scan. Click on the "+" sign next to each duplicate file found to see the location of the file on each drive.

There are newer versions of Comodo System Cleaner available though. Here is a guide to using the latest version to select drives to scan for duplicate files:

http://help.comodo.com/topic-42-1-148-812-duplicates.html

If you decide to use any of the other features I'd suggest creating another back up first as the cleaning features of the newest version can be a little aggressive if settings are changed from default.

Note: This post was edited by forum moderator to fix link on 03/04/2011 on 4:54 PM PT

Collapse -
Another Method
by chrisallenealing / February 25, 2011 9:29 AM PST
Collapse -
Download locations
by jgaasbeek / February 26, 2011 3:32 PM PST

I tried the download links, but both return a 404 error.

Collapse -
Here's a link for Comodo System-Clean on CNET Download.com
by Lee Koo (ADMIN) CNET staff/forum admin / March 4, 2011 6:30 AM PST
In reply to: Download locations
Collapse -
Question
by wtstandup / March 4, 2011 12:08 PM PST

I have a program that will show my duplicate files and I can read they're the same files it finds.

My question is if there is 2 or more files of anything is it safe to delete all but the one file? Getting rid of the duplicated files.

Collapse -
No, it is not "SAFE" to delete all but one file...
by brian_pletcher / March 5, 2011 12:47 PM PST
In reply to: Question

Unless you know exactly WHAT you are deleting, removing duplicate system files could render your system inoperable. IF you don't know what you are deleting, don't do it. If they are duplicate pictures, you are fine, but if they are duplicate exe or dll files or if you have the default windows settings that don't TELL you what you are deleting, it's risky business.

Collapse -
Getting Rid of duplicate files
by waynecz / February 25, 2011 9:49 AM PST

There is a free registry/privacy/disk cleaner software from Comodo called Comodo System Cleaner that under the "disk cleaner" section of the software has a duplicates tab. By clicking on the duplicates tab you can specify a drive, folder, etc. on what to check for duplicated files. The software then searches where you specified and anything that is duplicated it will show you the files that are duplicated (or triplicated, etc.) and allow you to pick which one of the files you want to delete. The one thing it cannot do is if you have the same file, let's say a picture, that is saved in different formats, GIF and JPG it cannot under that circumstance tell that they are the same picture. Everything else though it finds. I have used it and tested it to make sure the files were duplicate and it works like a charm. I'd suggest picking one folder first or drive that you know has duplicates and do a test run so you can see how it works. But it definitely DOES work. Plus, it is a fantastic, in my opinion, free registry, privacy, and disk cleaner software. And best of all, it's FREE. You can find it at this link - http://system-cleaner.comodo.com/ Hope that helps!

Collapse -
SpaceMan99
by ejamesr / February 25, 2011 10:06 AM PST

I've used SpaceMan99 found at extrabit.com. It identifies all duplicates, triplicates, etc. and let's you quickly jump to the file or folder where the duplicates reside.

Collapse -
Duplicate files in hardrive WD My Book and PC

I don't have a solution but I also need answers! My WD external drive seems to replace some of the files I deleted. I spent hours and days trying to organize it all but can't. I removed all the added crap I could when I bought external drive. Even my tech support guy did. I think it is multiplying files while I sleep. However some files have been lost or corrupted even though I try to be nice to it and not do anything I shouldn't Does anyone want to buy a 1 TB WD My Book??? Holds lots of junk,,,, Has mind of its own!
June (Win XP PRO- Custom built for video editing)

Collapse -
Yeah, I agree with you! ,but....
by Dumbpeni / February 25, 2011 1:53 PM PST

Why cant this be included in Windows??? Windows left a lot to be desired... Anyway, my idea about dealing with duplicates or "dummy" and tripilcates is to buy a huge hard drive to move all those files to then search the entire harddrive in one folder. Now you can get all files in order by name and start looking down the long, long,,long, long list of zillions of files and delete all the duplicates and triplicates in rows..
As matter of factly , I just bought a huge 500 gb hard drive that I am gonna do the project when I find time to do so.. I am staring at my brand new 500 gb drive and thinking "when, when?? {

Collapse -
Too many duplicated files in too many places
by junecareyart / February 25, 2011 2:19 PM PST

In answer to Dumbpeni: that was what I thought I bought the giant 500 MG My Book for... and for some reason the integral programs in the hard drive seeem to divide the files in such a way that make duplicates in more than one section. However I am not an expert, but my tech support did think he had removed all that junk but the ext hard drive actually put some of it back... on the desktop of my PC, no less. Maybe the thing is haunted.
I also bought the external hard drive to save movies I was in the process of editing with software by Roxio. However, Roxio refused to allow me to edit them without moving them back to the computer and re-registering each section. Maddening. I now have millions of segments of movies , thousands of photos,in my PC and also the hard drive and need software to help me deal with all the multiplying files. But, I have been unable to download any of the reccomended freewaree on CNET without getting other junk I am afraid of.

Collapse -
Interesting idea, and simple to boot! Even easier..
by H3lpdesk / April 25, 2011 1:35 PM PDT

In Win 7, create a library with all of the drives/folders that you want to check, and enter a mask in the search box (*.* if you want everything); that way, you can compare/tidy without having to copy files.

What I'm trying to find, is an easy way to find only the dupes, leaving the newest copy to trim archives, so that older ones only have unique files.

I use Tree Size Professional byu Jam Software, and it does a nice job of finding duplicates; and also has a fantastic graphical display of file allocation.

Popular Forums
icon
Computer Help 47,885 discussions
icon
Computer Newbies 10,322 discussions
icon
iPhones, iPods, & iPads 3,188 discussions
icon
Security 30,333 discussions
icon
TVs & Home Theaters 20,177 discussions
icon
HDTV Picture Setting 1,932 discussions
icon
Phones 15,713 discussions
icon
Windows 7 6,210 discussions
icon
Networking & Wireless 14,510 discussions

Tech for the holiday

Find recipes for July 4 with these foodie apps

The Fourth of July means fireworks, fun and food. If you're planning on a barbecue this weekend, we've got the apps to help you find holiday-inspired recipes.