It's not really 'of course' that it doesn't happen with text only mails. In fact, it's rather incomprehensible to me. Just to make things more clear, can you explain what you mean with 'text only'. Is it 'plain text formatted' or 'html-formatted with only text'?

The first thing to do is to ascertain if this is a problem on your side or with your ISP. Start with sending or forwarding such a mail to yourself and see what happens. Then open the properties of the mail and have a look at the email header to see if there's any notable difference compared with a 'text only' email.

It might be necessary to contact your ISP's helpdesk about this problem. Having such a mail yourself is a good preparation for that.