One of the aspects of security is controlling the environment. If I'm checking my email at home, I pretty much control the entire environment... everything on my computer, all hardware devices, and the network in general. If you ever check your email outside of that environment, like on your phone, at the office, at a public place using a public connection, you are now no longer in control. There are lots of ways people can gain access.

There could be a key logger that's not being detected, they could play a man in the middle attack, they could capture your cookies (without getting actual login credentials), if you use the same password somewhere else, they could have obtained it and used it to login into your email, they could have used the "I forgot my password" to reset your password by answering a couple simple questions, etc. There are too many variables and too many ways. The sad part is, if someone really wants to get into your email, unless you control the environment, a targeted attack can almost always be successful. Hence why email should not be used for anything sensitive.

I personally have several email accounts for different levels of sensitivity. The ones most sensitive I only use in places where I control the environment. You just have to find the write balance, because often security comes at the price of convenience.

~Sovereign