Duplicate music woes? Let iTunes Match clean up the mess!

Years ago, I went digital with my music. I ripped all my CDs, bringing everything into Windows Media Player. I didn't start with the iPod. The Rio Karma was my main MP3 player. Over the years, my music library moved from computer to computer. It migrated into iTunes (and into iPods, iPhones, iPads....). It gained new music from purchases or the occasional rare CD that I'd buy. And my music got really messy.

I shouldn't really care that somehow, I ended up with two or three different copies of the same songs. What does it really matter, especially as storage is so cheap these days? But it did to me. It seems to matter to others, too.

After all, iTunes has a "Display Duplicates" feature, plus there's no lack of programs out there offering to "clean" or "de-duplicate" music libraries. Heck, so many people search for ways to deal with duplicate songs that it's even a Google suggestion:

I've tried some de-duplication programs in the past and didn't come away impressed. Heck, I felt downright ripped-off by one. But happy days have arrived. I found a tool that really worked, Apple's iTunes Match service. It cleared out my duplicates plus gave me the additional benefits of cloud-based music and high-quality songs, all at the same time.

Let iTunes Match wash out pesky duplicates

iTunes Match came out last November. It checks your music library against songs that Apple sells. If there's a match, you can then download that song to another computer, mobile device or upgrade the song to the same computer, to a better quality than you might currently have.

iTunes match isn't free. It runs $24.99 per year. But that's around the price I've seen some duplicate cleaning tools charge. That alone made it worth the cost for me. I (or anyone) could pay for only one year, to do a one-time de-duplication. But I like the service enough I'll keep renewing for the other features.

To clean my library, I signed up for Match using iTunes on the computer that had my current library of music. That was a Windows PC, by the way. iTunes Match is a solution that works for both Windows and Mac. That computer had about 8,000 songs, of which around 1,800 were reported as duplicates (using the File > Display Duplicates feature in iTunes).

Some of these were duplicates for good reasons -- songs that happened to appear on more than one album, for instance. But others were songs that somehow got copied into my library twice or three times, possibly because of accidental imports over the years.

Duplicate free, in the iCloud

I let Match do its work over the course of a day or two. It scanned my library, then it uploaded songs as needed to create my library in the cloud. When it was done, I got a cheerful message telling me that my iTunes library -- with about 7,200 songs -- now lived in iCloud:

If you're doing the math, about 800 songs didn't make it from my computer into the cloud. I'll get back to that. But the next step in de-duplication was to pull my library back down from the cloud.

I have a separate computer that I use when traveling, a MacBook Air. I loaded up iTunes there, went to the Music area, then selected all the songs that were tagged as being in the cloud. I right clicked, selected "Download" and let the computer do its work.

Once the downloading was done, time for the test. Did I still have a duplicate problem? Nope!

Using the Display Duplicates feature, iTunes reported that I had about 300 duplicate songs, a huge drop from the 1,800 I originally started with. Most of these were "good" duplicates, where as I mentioned before, you get the same song but on different albums:

Hey, everyone needs three copies of "Blue Monday" by New Order! Actually, it would be nice if iTunes could let you download only a single copy of a song, if it's on multiple albums as with some of the examples above, and simply cross-link them behind the scenes.

Still, the main point is that almost all those "bad" duplicates that I had before were gone. From 1,800 duplicates, mostly bad, to only 300 good ones! I am still confused about how I started with 8,000 songs, lost around 1,500 duplicates yet still have 7,200 songs left rather than 6,500. I suspect that iTunes doesn't count duplicates correctly. But I do know that none of my music was lost, and I ended up with a nice, clean library. That leads to double-checking on iTunes.

Did iTunes Match do right by you?

If you do this, I think it's worth taking a close look after iTunes Match is done. I found a handful of cases where bad duplicates got through, the exact same song, on the exact same album, listed twice, such as with "OK Computer" from Radiohead:

In all, there were about 40 bad duplicates that I had to remove manually. Despite this, I was very happy with this tiny amount. iTunes succeeded in cleaning up my music library far beyond what I expected.

Here's a tip for manually checking on duplicates. Go to "Music" in your iTunes library, use File > Detect Duplicates. Then sort by song name, and you can easily scan to see if there are any bad duplicates of the same song, for the same album. If so, right click on one of the songs, select Delete, and you'll get an option to remove the song. Be sure to select remove from iCloud as well:

Do that, and the duplicate is gone forever. Apple has further instructions on deleting songs from iCloud here.

More verification and understanding iCloud Status tags

While I was pleased to have my duplicates gone, I wondered if iTunes, as part of the matching process, somehow lost some of my other songs. Nothing was lost, but there were a few songs that had errors or were ineligible for uploading, which needed to be dealt with manually.

After you sync your songs to iCloud, the computer you uploaded from will show an iCloud Status for each song. Go to "View" on the menu, select "View Options" and then tick the "iCloud Status" box. Tick the "iCloud Download" one, too, if it's not already selected.

After you've done this, you can sort all your music by iCloud Status. After my upload was done, about 5,900 of my songs had the status of "Matched." That meant these were songs that in my library that iTunes matched to songs it already had in the cloud. These songs were never pushed from my computer to iCloud, but my ownership of them was verified. That allows me to download any of them to any device I have that runs iTunes.

An important point here. Any songs that are matched, if you download them, come down fresh from Apple in 256 kbps quality and AAC format, even if they were originally in MP3 or another format. Apple explains more about this here. If you must have MP3 format, you'll need to convert the songs later. But many devices, including Apple's own, support AAC, so this shouldn't be an issue for many people.

About 1,200 songs were tagged as "Uploaded," which means iTunes didn't have matches for them in its library. Because of this, my computer uploaded them to iCloud. If I download them to another computer or back down to the same computer I uploaded from, I'll get back exactly what was uploaded.

That was the vast majority of my library. About 50 songs were tagged as "Purchased," which means I purchased them directly from iTunes and can download if necessary, as with Matched songs (I buy most of my music from Amazon, which is why my number of purchases may seem low).

Only about 15 songs got tagged as "Error," and in all of these cases, another copy of the song made it into iCloud, so there was nothing to worry about. Only five songs got tagged as "Ineligible," which meant I didn't have them in good enough quality for uploading to iCloud.

Macworld has a nice guide to iCloud status messages here, if you're looking for more guidance, and Apple also has a page about the iCloud status icons that you can also make appear.

Creating your new library

Obviously, you'll need to take the closest look at any of your music that's flagged as "Error" or "Ineligible." You'll have to find other ways to get these songs into your iTunes library, if you recreate it fresh by downloading from iCloud.

That leads to creating that fresh, new library. If you have a separate computer, as I did, then you can download straight from iCloud. You'll only get songs that on your original computer had a status of "Matched," "Uploaded" or "Purchased." When done, that computer will have a clean, de-duplicated music library.

If you have only one computer, and you want that computer to mirror your de-duplicated library that's in iCloud, the safest option is to create a new iTunes library (see Apple's instructions here). After you do this, and you've logged into iTunes Match, all your music will be listed with little cloud icons next to the songs, similar to this:

The icons mean the songs are available for download. Select all the songs in your library, right click and choose "Download." Now your library will be recreated, from iCloud, without all those duplicates you originally had. If you're happy with your new library, you can ignore the old one or delete it, eventually.

Alternatively, you could work from within your existing library. To do that, select all your music and delete. DO NOT TICK THE DELETE FROM ICLOUD OPTION WHEN YOU DO THIS. If you also delete from iCloud, there will be nothing to download to your computer. 

After you've done this, as above, all your music that had a "Matched," "Uploaded" or "Purchased" status should still be listed, with little cloud icons as shown above. That's what happened in my test of this. Select all your songs, then download.

If you work within your existing library, I recommend that to be super-safe, backup your iTunes library before doing the delete and download option.

One other benefit is that by de-duplicating, I now have a clean library I can use if I want to upload it elsewhere, such as Google Music or Amazon. By the way, I did wonder if Google Music might be a free way to do the same de-duplicating that iTunes Match does. I'm pretty sure it doesn't. I've uploaded some music there over the past year, and I can see that I have two or three copies of the exact same songs.

As I said, I was very happy with the iTunes Match process. I cleaned up my music library, plus I've enjoyed the freedom of not having to keep 30GB or more of music backed-up locally, since I have an ultimate backup in the cloud -- the iCloud, that is.