I've been using the Sony H.ear headphones. They have a clip that fits behind the ears and the actual speaker fits inside of the ear. Also, they sound fairly good and are not terribly expensive (a set should cost you about $10).
I've found them very comfortable (I can wear them for hours without discomfort) and they stay firmly in place until you remove them (when I've accidentlly knocked away the cord my player went flying from my hand before the headphones came off). During workouts they should stay in place regardless of what you are doing.
Looking for recomendations.
I've tried the Sony Yellow ones that fit into the ears. They won't stay on properly and they don't sound good either.
Ear buds don't stay in for me.
I now have some cheap Panisonic things, and they are better, but they rattle a bit (and that's annoying)
How about some of the less expensive noise cancellation headsets?
Your thoughts welcome.

Chowhound
Comic Vine
GameFAQs
GameSpot
Giant Bomb
TechRepublic