Listening to very high quality music will also drain your battery faster. What are your bitrates? Are you using MP3s?
I recently bought an 8GB Sansa Fuze and it rated the battery life at 24 hours. I've read a number of reviews and their drain tests go anywhere from 19 to 28 hours. Mine does not even get to twelve (I listen to my music at about 30%-40% volume). Is there something wrong with my battery?

Chowhound
Comic Vine
GameFAQs
GameSpot
Giant Bomb
TechRepublic