I have a number of friends who love music and have the worst audio systems, and though I've tried to help them out most couldn't care less about the sound of their music. I've known folks with massive LP and CD collections who were perfectly content with the sound of the crappiest of hi-fis and headphones. This diverse group included a smattering of recording engineers, musicians, and even record store owners!
Musicians tend to say things like, "I live with live music, why buy a decent set of headphones or speakers?" They'd rather buy instruments, or they claim poverty, and don't have extra money to buy audio gear.
True, those are all valid excuses, but even when I've offered my old gear, there are few takers. Sound quality just isn't a big deal for a lot of folks, and musicians claim they just listen to the music, not the sound of it, and some of them just listen to how well the music is being played. Guitarists zero in on the guitar parts, drummers listen to drummers, and vocalists notice everything about how a song is sung.
As any reader of the Audiophiliac knows, sound is a really big deal for me. I assume it's the same for most audiophiles: a great speaker or headphone conveys more than just notes, melody, harmony and rhythm. The texture of the distortion of Jimi Hendrix's guitar, the sadness in Billie Holiday's voice, the heartbeat rhythms of African drumming, the majestic swell of the New York Philharmonic orchestra, you only get to hear all of that with the best headphones and speakers. Sure, music is audible over free earbuds or your smartphone's speakers, but the weeks or months of hard work the band and engineers put into perfecting the sound of the music will go unheard.
I love sound just as much as music, when I walk down a city street the cacophony of sounds coming from every direction -- traffic, people talking, birds, the low pulse of an energized landscape -- all of that is endlessly fascinating to me. You guys living in rural settings have your own aural tapestries.
Basically, I'm for whatever it takes to enjoy recorded music at its best, and as I've pointed out from time to time there's no shortage of terrific, highly affordable gear to choose from, but if you're happy with earbuds, good for you!